Glen Mazza's Weblog

Main | Next page »

https://web-gmazza.rhcloud.com/blog/date/20141124 Monday November 24, 2014

Using UsernameToken security with Apache CXF

This tutorial modifies the CXF version of the WSDL-first DoubleIt web service to include WS-Security with UsernameTokens. This profile should be used with transport-layer encryption (i.e., SSL) as a minimum to ensure the username and password are encrypted at least between the client and the first recipient node. Note the X.509 or SAML token approach would be more appropriate instead if any of cases below apply:

  • Clients need to sign the request to guard against it being altered in transit
  • There are intermediary nodes between client and service that would be unencrypting the message (and hence able to read its contents) before forwarding it on to the next node
  • The web service provider has incomplete nonce and timestamp support necessary to guard against replay attacks. CXF has implemented such caching since version 2.6, and it also appears supported with Metro.

CXF provides two main options for adding UsernameToken security headers, both of which will be covered below: WS-SecurityPolicy and manual CXF interceptors. The former relies on the WSDL already having WS-SecurityPolicy elements defined within it to obtain the security requirements. Use the manual CXF interceptor approach when security is not defined in the WSDL or more customized control of the security header construction is desired.

Note one difference between CXF's UsernameToken implementation and Metro's is that the former requires explicit service-side callback handlers to validate passwords while the latter, for the Tomcat and the GlassFish server, will validate against container usernames and passwords if no service-side callback handler is declared. However, CXF provides a JAASLoginInterceptor for container-based authentication.

The finished tutorial source code can be obtained from GitHub by using either the download ZIP button or git clone -v git://github.com/gmazza/blog-samples.git command. The download is configured to use WS-SecurityPolicy, if desired make the adjustments specified below to switch to the CXF interceptor approach.

  1. Implement SSL without basic authentication for the web service. This is the same as Step #1 of the Metro/UsernameToken guide.

  2. Include additional Maven dependencies to handle security processing. The following dependency will need to be added to the project-level pom.xml to activate security processing.

    <dependency>
       <groupId>org.apache.cxf</groupId>
       <artifactId>cxf-rt-ws-security</artifactId>
       <version>${cxf.version}</version>
    </dependency>
    

    Also, since we'll be adding a Spring cxf.xml configuration file to the client to declare the security parameters, we'll need to add a Spring dependency to the client/pom.xml. (See notes at the bottom for a non-Spring way of doing client configuration.) While CXF is relatively flexible with the version of Spring that you use, to determine the optimal (tested) version of Spring for your version of CXF, go to the tags folder of the CXF source repository, select your version of CXF, and under that folder select the folder named "parent" and open its pom.xml. Search for the cxf.spring.version variable to determine the version to use.

    <dependency>
       <groupId>org.springframework</groupId>
       <artifactId>spring-context</artifactId>
       <version>${spring.version}</version>
    </dependency>
    
  3. Configure the client to provide the user and password. We'll be introducing a Spring-based cxf.xml configuration file, presently configured for the WS-SecurityPolicy approach but can be switched over to the CXF interceptor approach by following the comments given below in this source file.

    Next supply the password CallbackHandler that was listed in the SOAP client above. Place this class in the same package as the SOAP client:

    package client;
    
    import java.io.IOException;
    import javax.security.auth.callback.Callback;
    import javax.security.auth.callback.CallbackHandler;
    import javax.security.auth.callback.UnsupportedCallbackException;
    import org.apache.wss4j.common.ext.WSPasswordCallback;
    
    public class ClientPasswordCallback implements CallbackHandler {
    
        public void handle(Callback[] callbacks) throws IOException, 
                UnsupportedCallbackException {
            WSPasswordCallback pc = (WSPasswordCallback) callbacks[0];
    
            if ("joe".equals(pc.getIdentifier())) {
                pc.setPassword("joespassword");
            } // else {...} - can add more users, access DB, etc.
        }
    }
    
  4. Add the WS-SecurityPolicy statements to the WSDL. This has already been done in the Service WSDL in the source code download, which uses the WS-Policy statements created in the Metro UsernameToken tutorial. CXF interceptor method only: We'll need to use a Policy-free WSDL, so remove this DoubleIt.wsdl and rename the Policy-free DoubleItInterceptorMethod.txt found in the service submodule resources folder to DoubleIt.wsdl (it's the same WSDL from the standard DoubleIt tutorial.)

  5. Configure the service provider to authenticate the username and password tokens. Here we'll need to modify the cxf-servlet.xml configuration file from Step #6 of the WSDL-first tutorial to add either the CXF inbound interceptor or WS-SecurityPolicy configuration information (not both.) Replace the previous cxf-servlet.xml file with the following, after modifying it based on your security implementation preference:

    Once done we'll need to add the new server-side password callback class referenced above, which provides the correct password for a user, which the CXF framework uses to compare with what is supplied in the client request. Place the following ServerPasswordCallback class to the same package as the service's DoubleItPortTypeImpl SEI implementation class:

    package service;
    
    import java.io.IOException;
    
    import javax.security.auth.callback.Callback;
    import javax.security.auth.callback.CallbackHandler;
    import javax.security.auth.callback.UnsupportedCallbackException;
    import org.apache.wss4j.common.ext.WSPasswordCallback;
    
    public class ServerPasswordCallback implements CallbackHandler {
    
        public void handle(Callback[] callbacks) throws IOException,
                UnsupportedCallbackException {
            WSPasswordCallback pc = (WSPasswordCallback) callbacks[0];
    
            if ("joe".equals(pc.getIdentifier())) {
               pc.setPassword("joespassword"); 
            }
        }
    }
    

    You can now compile and redeploy the web service from the DoubleIt base folder.

  6. Test the client. Run the client as shown in the DoubleIt tutorial. Best to test also with incorrect usernames and passwords to ensure the web service provider is catching these errors.

    You can view the headers being added to the SOAP request if you temporarily switch your web service back to non-SSL usage. This can be done by updating the web.xml and URL within the WSDL, and additionally if you're using WS-SecurityPolicy, removing any WSDL element declaring a requirement for SSL. Once done you can either use Wireshark or activate logging within your WSClient class to see the unencrypted SOAP messages. Running the client from a terminal window will show results similar to the following:

    Oct 1, 2011 7:47:34 PM
    org.apache.cxf.interceptor.AbstractLoggingInterceptor log
    INFO: Outbound
    Message
    ---------------------------
    ID: 1
    Address:
    http://localhost:8080/doubleit/services/doubleit
    Encoding: UTF-8
    Content-Type: text/xml
    Headers: {Accept=[*/*], SOAPAction=[""]}
    Payload:
    <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
        <soap:Header>
            <wsse:Security
                xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"
                xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"
                soap:mustUnderstand="1">
                <wsse:UsernameToken wsu:Id="UsernameToken-1">
                    <wsse:Username>joe</wsse:Username>
                    <wsse:Password
                        Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText">joespassword</wsse:Password>
                </wsse:UsernameToken>
            </wsse:Security>
        </soap:Header>
        <soap:Body>
            <ns2:DoubleIt xmlns:ns2="http://www.example.org/schema/DoubleIt">
                <numberToDouble>10</numberToDouble>
            </ns2:DoubleIt>
        </soap:Body>
    </soap:Envelope>
    --------------------------------------
    Oct 1, 2011 7:47:34 PM
    org.apache.cxf.interceptor.AbstractLoggingInterceptor log
    INFO: Inbound
    Message
    ----------------------------
    ID: 1
    Response-Code: 200
    Encoding:
    UTF-8
    Content-Type: text/xml;charset=UTF-8
    Headers: {Content-Length=[238],
    content-type=[text/xml;charset=UTF-8], Date=[Sat, 01 Oct 2011 23:47:34
    GMT], Server=[Apache-Coyote/1.1]}
    Payload:
    <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
        <soap:Body>
            <ns2:DoubleItResponse xmlns:ns2="http://www.example.org/schema/DoubleIt">
                <doubledNumber>20</doubledNumber>
            </ns2:DoubleItResponse>
        </soap:Body>
    </soap:Envelope>
    --------------------------------------
    The number 10 doubled is 20
    

    If you do this, remember to switch back to SSL for production usage.

Notes

If desired, you can avoid introducing a Spring dependency in the client by not using a cxf.xml configuration file and instead declaring security using the Java API in the client's WSClient class. A sample alteration demonstrating both the WS-SecurityPolicy and CXF interceptor approaches is as follows:

...
import org.apache.cxf.endpoint.Client;
import org.apache.cxf.endpoint.Endpoint;
import org.apache.cxf.ws.security.wss4j.WSS4JOutInterceptor;
import org.apache.ws.security.WSConstants;
import org.apache.ws.security.handler.WSHandlerConstants;
...

public class WSClient {
   
    public static void main(String[] args) {
        DoubleItService service = new DoubleItService();
        DoubleItPortType port = service.getDoubleItPort();

        // WS-SecurityPolicy configuration method
/*
        Map ctx = ((BindingProvider)port).getRequestContext();
        ctx.put("ws-security.username", "joe");
        ctx.put("ws-security.callback-handler", ClientPasswordCallback.class.getName());
        // instead of above line can also do:
        // ctx.put("ws-security.password", "joespassword");
*/
        // Alternative CXF interceptor config method
/*
        Client client = org.apache.cxf.frontend.ClientProxy.getClient(port);
        Endpoint cxfEndpoint = client.getEndpoint();
        Map outProps = new HashMap();
        outProps.put(WSHandlerConstants.ACTION, WSHandlerConstants.USERNAME_TOKEN);
        outProps.put(WSHandlerConstants.USER, "joe");
        outProps.put(WSHandlerConstants.PASSWORD_TYPE, WSConstants.PW_TEXT);
        outProps.put(WSHandlerConstants.PW_CALLBACK_CLASS,
            ClientPasswordCallback.class.getName());
            
        WSS4JOutInterceptor wssOut = new WSS4JOutInterceptor(outProps);
        cxfEndpoint.getOutInterceptors().add(wssOut);
*/        
        ...

https://web-gmazza.rhcloud.com/blog/date/20141006 Monday October 06, 2014

Returning PDFs from Web Services using MTOM and Apache FOP

Summary: Use MTOM and Apache FOP within your Apache CXF- or Metro-based web services to send PDFs to and from SOAP clients.

[Read More]

https://web-gmazza.rhcloud.com/blog/date/20140929 Monday September 29, 2014

Activating Transport Layer Security (SSL) for web services

See how to implement SOAP web services over SSL.

[Read More]

https://web-gmazza.rhcloud.com/blog/date/20140922 Monday September 22, 2014

Creating a Java-first web service using CXF or Metro

Summary: This article shows the changes needed to the WSDL-first DoubleIt tutorial to convert it to a start-from-Java (no WSDL) process.

[Read More]

https://web-gmazza.rhcloud.com/blog/date/20140915 Monday September 15, 2014

Creating a SOAP client with either Apache CXF or GlassFish Metro

Summary: This tutorial shows how SOAP clients can be created using either Apache CXF or GlassFish Metro. We'll be creating a Maven-based client for the eBay Shopping Web Service, which provides a full range of search and retrieval capabilities for items being auctioned on the eBay site.

[Read More]

https://web-gmazza.rhcloud.com/blog/date/20140908 Monday September 08, 2014

Creating a WSDL-first web service with Apache CXF or GlassFish Metro

Summary: This tutorial shows how to create a WSDL-first web service using either Apache CXF or GlassFish Metro. We'll be using Maven as our build tool, with web service deployment on either Tomcat (for CXF or Metro) or OSGi deployment with Apache Karaf (CXF only).

[Read More]

https://web-gmazza.rhcloud.com/blog/date/20140614 Saturday June 14, 2014

Creating a Java Swing alternative to JConsole for calling MBeans

In this article I show how to create a Java Swing application to interact with MBeans in a manner specific for your needs, reducing need to use JConsole.

[Read More]

https://web-gmazza.rhcloud.com/blog/date/20140427 Sunday April 27, 2014

Working with Apache Roller source code

This entry explains the development environment and process I use for working on the Java-based Apache Roller blog server. I'm blogging this article to make it easier for those wishing to help with Roller development, also to assist Roller users wishing to modify Roller trunk code to better suit their needs (perhaps to bring in additional themes beyond those pre-packaged by Roller.) I use IntelliJ IDEA community edition on Ubuntu, but other IDEs and OSs are also used by other Roller team members. Some prerequisites for programming Roller:

ToolBefore proceeding, make sure you can...
JDK 1.7 or 1.8Activate "java" and "javac" from any folder in a command-prompt window
Apache MavenActivate "mvn" from any folder in a terminal window
Command-line Subversion clientRun "svn" from any folder in a terminal window
Standalone Tomcat 7 or 8Can start and view Tomcat from http://localhost:8080
MySQL or Apache Derby databaseCan activate mysql or ij (for Derby) from any folder in a command-prompt window.
Firefox Web BrowserOptional; used for running Selenium tests without needing to activate Tomcat
SQL Query tool such as SquirrelSQLOptional but recommended; used for querying the Roller database instance.

Trying to keep this blog of reasonable size, if you're new to any of the above tools their usage and setup are all well documented elsewhere on the 'Net. If you're already a Java developer and have GUI alternatives for the above command-line tools they should work fine as well (but be mindful of Ben Franklin's admonition about deviating from the terminal window for your work. :)

The instructions here are for Roller trunk (upcoming 5.1 version), which is much simpler, faster to build, and more modernized compared to the Roller release (5.0.x) versions; most all development is now occurring on the trunk.

  1. Check out Roller source and add any desired additional themes - Do an SVN checkout of Roller trunk and build it via the standard Maven mvn clean install command. If you wish to add any additional blog themes, place them in the src/main/webapp/themes folder prior to building Roller. Make sure you have a successful Roller build (including JUnit tests) before proceeding further.

  2. Prepare database and (optional) mail configuration - Follow the short Chapter 5 and Chapter 6, Sections 1-4 of the Roller Install Guide (ODT). Chapter 5 provides the few commands needed to configure an empty MySQL, PostgreSQL, or Derby database and Chapter 6 the configuration of the roller-custom.properties file, optional Mail configuration, and the required Mail and JDBC JARs that will be needed in Tomcat's lib folder. The test roller-config.properties file that I have in my Tomcat lib folder is as follows:

    # Any properties placed below overrides Roller default settings defined in
    # the roller.properties file embedded within the Roller WAR, and should be 
    # stored in the CATALINA_HOME/lib folder for Tomcat.
    
    installation.type=auto
    mediafiles.storage.dir=/myfiles/roller-inst/mediafiles
    search.index.dir=/myfiles/roller-inst/searchindex
    
    # Roller file and logging level
    log4j.appender.roller.File=${catalina.base}/logs/roller-tomcat.log
    log4j.logger.org.apache.roller=INFO
    
    #MySQL
    database.configurationType=jdbc
    database.jdbc.driverClass=com.mysql.jdbc.Driver
    database.jdbc.connectionURL=jdbc:mysql://localhost:3306/rollerdb?autoReconnect=true&useUnicode=true&characterEncoding=utf-8&mysqlEncoding=utf8
    database.jdbc.username=myusername
    database.jdbc.password=mypassword
    
    # EclipseLink debugging
    eclipselink.logging.file=/media/work1/underthehood/apache-tomcat-7.0.29/logs/eclipselink-tomcat.log
    eclipselink.logging.level=FINER
    
    # Mail config (See Roller Install Guide)
    mail.configurationType=jndi
    mail.jndi.name=mail/Session
    

    Before moving on to the next step, best to deploy the app/target/roller.war to your Tomcat webapps directory, start Tomcat and confirm you can run the application at http://localhost:8080/roller. This is an important one-time check to ensure your database, roller-custom.properties file, Tomcat, and Roller WAR are all properly configured, and once confirmed you should be in good shape for all subsequent coding and debugging. If any deployment problems, make sure you've reviewed Chapters 5 and 6 of the Install Guide, and if necessary contact the Roller User's List for assistance.

  3. Create a script to start development with everything needed - For efficient start-up I've created a hackroller.sh script that opens up all needed windows and applications at once instead of having me needing to do so manually each time I start development. A simplified version of my hackroller.sh is as follows and explained below:

    hackroller.sh:

    gnome-terminal --geometry=132x24 \
       --tab-with-profile=HasTitle --title "ROL Trunk" --working-directory="/work/opensource/roller-trunk" -e "bash -c \"svn update; exec bash\"" \
       --tab-with-profile=HasTitle --title "ROL Trunk2" --working-directory="/work/opensource/roller-trunk" -e "bash -c \"echo -Dmaven.surefire.debug; exec bash\"" \
       --tab-with-profile=HasTitle --title "Servlet Container" \
       --tab-with-profile=HasTitle --title "IntellijIDEA" -e "bash -c \"sh idea.sh; exec bash\"" \
       --tab-with-profile=HasTitle --title "GEdit" -e "gedit worklog.txt /home/gmazza/hackroller.sh $CATALINA_HOME/lib/roller-custom.properties $CATALINA_HOME/logs/catalina.out $CATALINA_HOME/logs/roller-tomcat.log `find $CATALINA_HOME/logs/localhost*.log` $CATALINA_HOME/logs/eclipselink-tomcat.log /work/opensource/roller-trunk/app/target/roller.log /work/opensource/roller-trunk/app/target/eclipselink.log " \
       --tab -e "dolphin --geometry=600x500+1+1 /work/opensource/roller-trunk $CATALINA_HOME" \
       --tab -e "firefox http://roller.apache.org/ https://issues.apache.org/jira/browse/ROL http://localhost:8080/roller http://svn.apache.org/viewvc/roller/ https://analysis.apache.org/dashboard/index/org.apache.roller:roller-project"
    #  --tab-with-profile=HasTitle --title "SquirrelSQL" --working-directory="/work/tools/squirrel-sql-3.5.2" -e "bash -c \"sh squirrel-sql.sh; exec bash\""
    

    The above script opens a multitab Console window which in turn opens a few separate application windows. In particular:

    • I open two terminal tabs pointing to the Roller trunk branch. This allows me to build Roller from one terminal while possibly reviewing my changes (using commands svn status and svn diff, for example) in the second window. I echo the -Dmaven.surefire.debug in the second tab just to quickly remind myself the Maven string to add during JUnit test debugging (covered below).
    • I have a separate "ServletContainer" tab open to my home folder for manual activation of servlet containers (Roller is tested on Tomcat, GlassFish, and JBoss) or other applications from the command line. I don't need this as often due to the copy.sh script (discussed below) I now use.
    • I use a tab for starting IntelliJ IDEA.
    • I open the multi-tab GEdit text editor with several files that are handy for me during development:
      • worklog.txt - A scratchpad file of TODO notes, commands, etc. that I keep for personal use between coding sessions.
      • hackroller.sh - The very script above, I keep handy in case I need to make adjustments or additions to it.
      • roller-custom.properties - The Roller configuration file discussed above and used by all Roller deployments to my standalone Tomcat. Changes I made here are activated on the next Tomcat restart.
      • catalina.out, roller-tomcat.log, localhost*.log, eclipselink-tomcat.log - These are logging files filled by Tomcat and/or Roller during running, usually very important for debugging as error logging usually gets written to one of these files. When you see terse "System" or similar errors reported by Roller in the web browser be sure to check these files for the error in detail.
      • roller.log, eclipselink.log - These files are under Roller's target directory and are populated during JUnit test running via the Maven test phase, as discussed below.
    • Dolphin - A multitab file browser that I have opened to the Roller trunk code and Tomcat folders for easy access to files.
    • Firefox - Here I pre-open tabs to the Roller website, JIRA task tracker and Sonar issues list, source code repository, and default location for Roller once it's deployed to Tomcat.
    • SquirrelSQL - Commented-out by default but used when I wish to query the Roller database to see the values that Roller is reading/writing. SquirrelSQL will need your database's JDBC JAR added to it as well as have the connection information you configured in roller-custom.properties.

    For scripts such as the above make sure there's no whitespace after the ending "\" on each line (error messages will pop up otherwise), and use a leading # (as shown above for SquirrelSQL) for actions you wish to disable by default. The $CATALINA_HOME specified above, as usual, is the base folder for your standalone Tomcat installation. In addition to the above configuration, you may wish to keep handy the Roller trunk documentation, kept in OpenOffice format in the Roller/docs folder or online.

  4. How to build and deploy to local Tomcat - After a successful mvn clean install from the Roller trunk folder, I run this copy.sh shell script that I keep in that folder:

    fuser -k 8080/tcp 8009/tcp 8005/tcp
    rm -r /media/work1/underthehood/apache-tomcat-7.0.29/webapps/roller
    cp ./app/target/roller.war $CATALINA_HOME/webapps
    rm $CATALINA_HOME/logs/*.log
    rm $CATALINA_HOME/logs/catalina.out
    $CATALINA_HOME/bin/startup.sh
    

    For rapid iteration when I don't need to run the tests each time, I simplify my build and deploy process to a single line: mvn clean install -Dmaven.test.skip ; sh copy.sh.

    The above file first kills the Tomcat instance (via fuser -k), deletes the previous expanded Roller webapp directory on the Tomcat instance and copies the latest created Roller WAR over, and clears all the logs before finally restarting Tomcat with the new WAR. The new Roller will be accessible at http://localhost:8080/roller again using the same $CATALINA_HOME/lib/roller-config.properties and database (i.e., all database-stored information including test blog data created will be immediately picked up by the new Roller WAR.)

  5. How to debug on local Tomcat - Learning to debug from your IDE any webapp running locally on standalone Tomcat is frequently vital when troubleshooting and thankfully simple to do. First, add to your Tomcat CATALINA_OPTS environment variable:

    export CATALINA_OPTS=$CATALINA_OPTS" -Xdebug -Xrunjdwp:transport=dt_socket,address=5005,server=y,suspend=n"
    

    Then from IntelliJ IDEA (or Eclipse), go to menu item Run -> Edit Configurations, add a Remote Configuration with a name of your choice and accepting the given defaults, and select OK. If Roller is already running on Tomcat, debugging can be activated at any time by setting breakpoints in the code that the Roller app would be activating, selecting menu item Run --> Debug {debug config name}, and then proceed with any needed code tracing.

  6. How to debug unit tests - To activate a specific Roller JUnit test, navigate to the {Roller Trunk}/app folder and run mvn test -Dtest=TestClassName#OptionalTestMethodName, as explained in the Maven Surefire Plugin documentation. To debug (code trace) the JUnit test within your IDE, add the -Dmaven.surefire.debug setting, set breakpoints in your IDE within code called by the tests and have it listen to port 5005 as before.

    Files to read: When running the JUnit tests, besides the Surefire results in the target/surefire-reports folder the target/roller.log file provides logging of the temporary in-process Roller instance. Also, to determine any potential SQL/JPA problems, EclipseLink JPA logging can be activated, with the output file also in the target folder, by uncommenting the eclipselink.logging.* properties in the app/src/test/roller-custom.properties file.

  7. How to run Selenium tests - Roller's Selenium tests test very basic functionality (setting up a blog and saving a blog entry) but can sometimes serve as a quick sanity check that minor, last-second source code changes did not cripple the application. Navigating to the trunk/it-selenium folder and running mvn clean install will cause a temporary instance of Roller to activate that is subsequently used by Selenium to run its tests in Firefox. For these tests, none of the database or Tomcat configuration created above will be used, instead Roller will be running an embedded Jetty and a temporary (in-memory) Derby database, and shut down once Selenium is finished.

Additional Notes:

https://web-gmazza.rhcloud.com/blog/date/20140203 Monday February 03, 2014

Creating Selenium tests for Java Web Applications

To reduce the amount of manual testing needed for the Java-based Apache Roller blog server, I added a Maven submodule that uses Selenium for automated in-browser testing. Presently only basic Roller functionality is being checked (create a user, create a blog, blog an entry, confirm the entry was saved), but I expect it to be filled out more over time. Its structure may be useful for other Java projects wishing to incorporate Selenium testing. The submodule POM relies on the Jetty Maven plugin to activate Roller, Brian Matthew's inmemdb-maven-plugin to activate an in-memory (i.e., no files created) Derby database instance, and finally the Maven Failsafe plugin to activate the Selenium tests, necessary as the tests run under Maven's integration-test phase.

To see Selenium in action, testing Roller (requires Firefox and Maven 3.0.5):

  1. Check out the Roller source using SVN:

    svn co http://svn.apache.org/repos/asf/roller/trunk roller_trunk
    
  2. Run mvn clean install from the roller_trunk (base) folder to build Roller and have it installed in your local Maven repository (from where it will be read by the Selenium tests). Building itself is quick (about two minutes on an average machine), however the initial download of Roller's dependencies, if not already in your Maven repo, could take some additional time.

  3. Navigate to the roller-trunk/it-selenium folder and run mvn clean install or mvn integration-test. Selenium will activate Firefox at Roller's home URL (http://www.localhost:8080/roller) and run its tests.

Some notes on creating Selenium-driven tests for web applications:

Using Selenium IDE to generate browser actions. Reviewing the nicely succinct documentation for Selenium IDE and Selenium WebDriver is a great way to get started. Selenium IDE is a Firefox plugin that records manual interaction with the application under testing ("AUT", using the documentation's terminology) into a script, which can then be activated from Selenium IDE to automatically re-run the same actions against the AUT. After adding testing assertions and verifications and confirming the script is moving through the AUT successfully, Selenium IDE can then be used to export the script as Java to incorporate into your WebDriver-backed Maven submodule. After becoming acquainted with the WebDriver Java API by working with a few exported files, you'll most probably find yourself able to code additional tests in Java directly without need for Selenium IDE.

Making adjustments to Selenium IDE scripts. Due to the manner in which Selenium IDE populates HTML form fields (perhaps by direct manipulation of the underlying HTML DOM document), certain mouse, key, and focus DOM events are not activated as they would be with manual data entry, resulting in necessary JavaScript not getting activated. For example, a submit button which would become enabled via JavaScript as a result of the data entry fields all being filled manually may remain disabled when Selenium IDE populates the form, making it unable to click that button and proceed. This occurred on one of Roller's registration screens -- the solution was to look for the DOM event in the JSP or generated HTML source which is needed to trigger the necessary JavaScript:

<tr>
...
    <td class="field"><s:password name="bean.passwordConfirm" size="20" maxlength="20" onkeyup="onChange()" /></td>
...
</tr>

<script type="text/javascript">
function onChange() {
    var disabled = true;
    var openIdConfig    = '<s:property value="openIdConfiguration" />';
    var ssoEnabled      = <s:property value="fromSso" />;
...

...and then add a fireEvent command within Selenium IDE prior to the command for clicking the Submit button:

Command:  fireEvent
Target:   id=register_bean_passwordConfirm
Value:    keyup

Note the Value above is keyup and not onkeyup; also, the target ID can be determined by having the browser display the HTML source for the page.

Exporting test cases (or a suite of test cases) into Java. Note that the Java exported cannot be reimported back into Selenium IDE for subsequent modification, although you can always save another copy of the test cases as HTML, load it into Selenium IDE for tweaking, and then do another export into Java. Also, exporting into Java is not strictly required (the Selenium Maven plugin used by Apache JSPWiki as shown here can work with Selenium IDE's default HTML), although I would not recommend HTML as you'll lose significant object-oriented coding advantages including code reuse.

The Selenium IDE File-->Export Test Case menu item provides three JUnit 4-based options:

  • RC - Uses the older Selenium 1 RC API.
  • WebDriver Backed - Uses Selenium 2's WebDriver to implement the Selenium 1 RC API. Good for transitioning from Selenium 1 to 2.
  • "pure" WebDriver - Uses Selenium 2's WebDriver API. I exported using this option, as presumably all new work should be based on Selenium 2.

In looking at the exported Java class(es), you may see commented "errors" about fireEvents (and possibly other commands) being unsupported, for example:

// ERROR: Caught exception [ERROR: Unsupported command [fireEvent | id=xxxx | keyup]]

This is usually not cause for alarm--the Selenium team decided not to support fireEvents in Selenium 2, feeling that WebDriver should instead internally fire the events that would occur if the data was entered manually. Alternatively, in certain cases testers can add actions that will cause those events to naturally activate. In my particular instance with the Roller submit button, it turned out no replacement coding was necessary as WebDriver, unlike Selenium IDE, was able to automatically fire the needed events based on the fields it filled. Note, worst case, it remains possible to execute JavaScript to fire the DOM events manually if the Java tests will not work otherwise, but before doing so, best to Google and/or search the Selenium Users Group with the specific "Unsupported command" message to see if a more standard solution is available.

Examine better ways to design tests. When working with the Java test classes, ways to improve their design using standard object-oriented techniques will become apparent. Foremost is moving to the Page Object design pattern (links). Thomas Sundberg's article shows the natural process of getting to that pattern by way of factoring out common functionality from the tests and additionally suggests using Cucumber for behavior-driven development. Some other suggestions:

  • Create an abstract base class for your page objects to handle common functionality--populating fields, validating screen titles, taking screen snapshots or logging the page source for errors, etc.

  • Although using the WebElement.click() method on form submits will normally halt processing until the next screen appears (and so far has always worked for me), the FluentWait object can also be used to explicitly halt Selenium until a specified HTML element on the new page appears (or a timeout you specify occurs).

  • For your page objects, create an additional multi-parameter constructor to allow for convenient creation of page objects in cases where the page is just being used to get to another page that is under testing. As such a page being activated with this constructor would not be under testing itself, just providing the minimum number of parameters in the constructor necessary to navigate to and test the desired page should be sufficient.

  • For time, accuracy, and efficiency, I would advise against turning your page objects into POJO's, with instance variables for each screen field and getters and setters for all fields. So far, I've added getters and setters for a field only when such a method is needed by a test case. Further, I'm not creating member variables in the page object for each widget, both to simplify the objects and out of fear that their values might deviate from what's actually on the browser screen. Instead, each accessor directly reads from or writes to the browser screen.

  • If you do wish to go the POJO route, take a look at the PageFactory object and @FindBy annotation, both described well on the ActivelyLazy blog.

  • In the Page Object model, when a submit button always moves the application from Screen A to Screen B, a typical Page Object method that your test classes will call will be as follows:

    public class LoginPage {
    ...
        public UserDashboardPage loginToApp() {
            // clickById() provided by AbstractRollerPage superclass
            clickById("login");  
            return new UserDashboardPage(driver);
        }
    ...
    }
    

    What do you do, however, if the subsequent screen could vary depending on the state of the application--for example, a login page might take you to a password-has-expired screen, a message notification screen if messages present, or the usual application screen if neither of the other cases hold? According to this article, it's recommended to have the page object implement different methods based on each different output possible, and have the test case call the appropriate method it's expecting based on the application state that it has created:

    public class LoginPage {
        ...other method above...
    
        public ChangePasswordPage loginToAppPasswordExpired() {
            clickById("login");
            return new ChangePasswordPage(driver);
        }
    
        public UserNotificationPage loginToAppUrgentNotification() {
            clickById("login");
            return new UserNotificationPage(driver);
        }
    ...
    }
    
  • Typically the constructor of a Page Object includes a sanity check verification that the page title is as expected (i.e., the WebDriver is actually on the page it is presumed to be on), throwing an exception if it is not. If you have multiple screens with the same title an alternative check based on an HTML element ID can be done. This will require that each page sharing the same title has a unique HTML ID attribute on an HTML element present only on that page, so you may need to have the page markup modified to include such an attribute.

Other Notes:

  1. To run multiple iterations of the same Selenium tests under different circumstances (e.g., using different security authentication methods), Juan Pablo of the Apache JSPWiki Team developed a WAR overlay method along with parameter filtering to configure each of the tests - check the JSPWiki IT Tests module to see the process.

  2. Functional Automated Testing Best Practices with Selenium WebDriver - presentation by Ben Burton

https://web-gmazza.rhcloud.com/blog/date/20140123 Thursday January 23, 2014

Advisement on traveling to Montreal/Toronto via New York with Greyhound Bus

I traveled to Montreal this past weekend via Greyhound from Washington DC (photos). For added flexibility, although it's a bit more expensive, I wanted to buy two one-way tickets instead of a round-trip, with the second purchased while I was in Canada, on the day I'm planning on leaving. This way, if there's a snowstorm in Canada I can leave a day later or earlier as necessary to escape blocked roads, or I can leave earlier or later in the day depending on my mood, etc., giving me much more flexibility. Buying separate tickets has not been a problem for me in the past, as three of my past four trips to Canada (one via MegaBus to Toronto and two via flight to Ottawa) I didn't buy a return ticket until I was in Canada.

Travelling to Montreal, the Washington to New York segment was fine but Greyhound refused to allow me to board the New York-to-Montreal segment because I didn't have a pre-bought return ticket. According to the lady taking my ticket at New York, validated by two other employees there, U.S. passport holders must show a pre-bought return ticket prior to being allowed to board. I was told this in stunned disbelief at 11:48pm of a 12:01am bus trip, for which the next available bus would be at 8:30am the following morning. After scrambling around the bus terminal trying to find where in the huge station I could buy Greyhound tickets, I was able to buy a return ticket just after midnight at the station, and then rush back to the gate. Thankfully, due to bus delays I was able to board and continue my trip, but this was a traumatizing 20-25 minutes for me that could easily have become an 8.5 hour wait in the bus terminal for the next bus if I couldn't buy a return ticket in time. Further, as my ticket to Montreal was print-at-home, it could have been declared invalid for the 8:30am ride, requiring me to buy yet another ticket north--however the Greyhound employee told me they probably would honor my 12:01am ticket for a later, same-day ride.

There is apparently an exception for U.S. passport holders with a visa indicating that one can stay forever in Canada, but that might be dependent on the Greyhound employee taking your ticket. Also, the lady taking my ticket had indicated the return ticket of a plane, train, or any company's bus ticket would be sufficient. Further, for simplicity, I could have bought the ticket right there on a smartphone, and shown it to her that way. If planning on driving back from Canada, one Greyhound ticket employee recommended buying a refundable ticket for the return. But, again, the precise terms may vary based on the individual Greyhound agent taking your ticket. Indeed, speaking with the Greyhound ticket agent at the intermediate Albany, NY station, he indicated his station at least does not have a round-trip purchase requirement at all. There seems to be too much variance on precise boarding requirements, and the Greyhound website does not mention the need to have a pre-bought return ticket, so I just wanted to blog my experience to help others avoid my experience. As two one-way tickets, one station-bought and hence more expensive, ended up costing much more than a round-trip ticket ($196 vs. $135) consider getting the latter to save the hassle and expense of a last-minute station purchase. Also good to confirm your tickets and passport with a Greyhound employee once you get to New York as soon as possible to confirm that you'll have no last-second problems boarding.

I felt Greyhound should have clearly stated in its "Traveling to Canada" section on its website that travelers with U.S. passports going through NYC must have a return ticket prior to being allowed to board a bus going to Canada. Had I known that, I would have happily bought the round-trip to begin with. I called Greyhound customer service and spoke with two levels of support there, but ultimately could not get a $60 refund of my additional costs incurred, as they felt the text in that section was already clear enough about the need to produce a pre-bought return ticket prior to boarding and they don't have a process for partial refunds of the type I was asking anyway. However, the 2nd level help said she might put in a suggestion to have the travel requirements more clearly stated on the website. Until then, incoming Googlers have my blog entry here. :)

Having all non-Canadian passengers possess a return ticket is not necessarily a bad thing--a bus cannot continue past the border until all on the bus are cleared, or those unclearable taken off the bus and sent to the adjacent U.S. customs side. Having a return ticket helps confirm to Canadian customs that you will be leaving their country, hence reducing the number of people sent to secondary and helping speed up the trip for all on the bus. Also, possession of a return ticket prevents disasters of people being refused entry and stuck at the Canadian border, 360 miles north of NYC, as it guarantees they will have a return ticket that they can use to get on a Greyhound going back home.

Outside of this hiccup, I must say, the drivers drove safely and efficiently, both going there and coming back, the buses were cozy and some of them were downright luxurious (leather seats!) and the WiFi worked well. Arguably, they did such a good job they deserved the extra $60. Other notes: for web purchases, tickets picked up via the Will Call option are generally much easier to exchange than the print-at-home ones possibly due higher chance of fraud the latter present, and Greyhound has tightened up its ticket exchange rules so make sure you well understand the ticket exchange policy on its website prior to buying any tickets when you are unsure about when you really want to go and come back. Also, although secondary inspections are infrequent, expect your belongings including laptops to be fully searched when you cross the border in either direction, so don't carry anything of an embarrassing or inappropriate nature.


Valid HTML! Valid CSS!

This is a personal weblog, I do not speak for my employer.