• Reusing Reducer Logic
• having multiple instance of same reusable redux react components on the same page/route

The problem description

Say in a web app, we have a lot of data tables on different pages, every table has a pagination component.

How do we setup current page index and current page size for each table? At first thought we might want to setup action creators and reducers for each of the page with the same methods and state inside, just with different module names.

// Action creator
const setPageIndex = pageIndex => ({
    type: 'SET_PAGE_INDEX',
    payload: pageIndex,
});

// Reducer
const createInitialState = () => ({
    pageIndex: 0,
});

const paginationReducer = (state = createInitialState(), action) => {
    switch (action.type) {
        case 'SET_PAGE_INDEX':
            return Object.assign({}, state, { pageIndex: action.payload });
        default:
            return state;
    }
};

But this way we’ll get a lot of duplicate code!

Namespace to the rescue

All these action creators and reducers share the same logic, just with different names, so in order to reuse the same logic, we should instead build action factory and reducer factory like the following.

Action creator actions/paginationActions.js:

const ACTION_TYPES = {
    SET_PAGE_INDEX: 'SET_PAGE_INDEX',
};

const setPageIndexFactory = namespace => pageIndex => ({
    type: `${namespace}/${ACTION_TYPES.SET_PAGE_INDEX}`,
    payload: pageIndex,
});

const actionsFactory = namespace => ({
    setPageIndex: setPageIndexFactory(namespace),
});

export { ACTION_TYPES, actionsFactory };

Reducer reducers/paginationReducer.js:

import { ACTION_TYPES } from 'actions/paginationActions';

const createInitialState = () => ({
    pageIndex: 0,
});

const paginationReducerFactory = namespace => (state = createInitialState(), action) => {
    switch (action.type) {
        case `${namespace}/${ACTION_TYPES.SET_PAGE_INDEX}`:
            return Object.assign({}, state, { pageIndex: action.payload });
        default:
            return state;
    }
};

export default paginationReducerFactory;

Say the namespace of one of our web pages is DOCUMENT/DETAILS, which is defined in constants.js:

const NAMESPACE = {
    DOCUMENT_DETAILS: 'DOCUMENT/DETAILS',
};

export { NAMESPACE };

In our reducers/index.js, define rootReducer as the following:

import { NAMESPACE } from 'constants';
import paginationReducerFactory from 'reducers/paginationReducer';

export default {
    [`pagination_${NAMESPACE.DOCUMENT_DETAILS}`]: paginationReducerFactory(NAMESPACE.DOCUMENT_DETAILS),
};

And here is how to use it in our components:

import React, { Component } from 'react';
import { bindActionCreators } from 'redux';
import { connect } from 'react-redux';

import { NAMESPACE } from 'constants';

import { actionsFactory as allPaginationActionsFactory } from 'actions/paginationActions';

const MODULE_NAMESPACE = NAMESPACE.DOCUMENT_DETAILS;

class DocumentDetails extends Component {
    render() {
        const { pageIndex, paginationActions } = this.props;

        return (
            <DataTable pageIndex={pageIndex} onPageIndexChangeHandler={paginationActions.setPageIndex} />
        );
    }
}

const mapStateToProps = state => ({
    pagination: state[`pagination_${MODULE_NAMESPACE}`],
});

const mapDispatchToProps = dispatch => ({
    paginationActions: bindActionCreators(allPaginationActionsFactory(MODULE_NAMESPACE), dispatch),
});

export default connect(mapStateToProps, mapDispatchToProps)(DocumentDetails);

Make it more universal

Say our data table component is universal, here is how to define it in a universal way:

import React, { Component } from 'react';
import { bindActionCreators } from 'redux';
import { connect } from 'react-redux';

import { actionsFactory as allPaginationActionsFactory } from 'actions/paginationActions';

class CommonDataTable extends Component {
    render() {
        const { pageIndex, paginationActions } = this.props;

        return (
            ...
        );
    }
}

const mapStateToProps = (state, ownProps) => ({
    pagination: state[`pagination_${ownProps.namespace}`],
});

const mapDispatchToProps = (dispatch, ownProps) => ({
    paginationActions: bindActionCreators(allPaginationActionsFactory(ownProps.namespace), dispatch),
});

export default connect(mapStateToProps, mapDispatchToProps)(CommonDataTable);

So in our DocumentDetails page we shall use it like this:

import { NAMESPACE } from 'constants';

const MODULE_NAMESPACE = NAMESPACE.DOCUMENT_DETAILS;

class DocumentDetails extends Component {
    render() {
        return (
            <CommonDataTable namespace={MODULE_NAMESPACE} />
        );
    }
}

Summary

Official Redux documentation express a different way but I feel the above approach is more comprehensive to me.

At last, reusing reducer logic is IMPORTANT simply because “DO NOT REPEAT YOURSELF” is IMPORTANT.

Refs and the Reflog

Git is all about commits: you stage commits, create commits, view old commits, and transfer commits between repositories using many different Git commands. The majority of these commands operate on a commit in some form or another, and many of them accept a commit reference as a parameter. For example, you can use git checkout to view an old commit by passing in a commit hash, or you can use it to switch branches by passing in a branch name.

By understanding the many ways to refer to a commit, you make all of these commands that much more powerful. In this chapter, we’ll shed some light on the internal workings of common commands like git checkout, git branch, and git push by exploring the many methods of referring to a commit.

We’ll also learn how to revive seemingly “lost” commits by accessing them through Git’s reflog mechanism.

Hashes

The most direct way to reference a commit is via its SHA-1 hash. This acts as the unique ID for each commit. You can find the hash of all your commits in the git log output.

commit 0c708fdec272bc4446c6cabea4f0022c2b616eba
Author: Mary Johnson <mary@example.com>
Date:   Wed Jul 9 16:37:42 2014 -0500

    Some commit message

When passing the commit to other Git commands, you only need to specify enough characters to uniquely identify the commit. For example, you can inspect the above commit with git show by running the following command:

git show 0c708f

It’s sometimes necessary to resolve a branch, tag, or another indirect reference into the corresponding commit hash. For this, you can use the git rev-parse command. The following returns the hash of the commit pointed to by the master branch:

git rev-parse master

This is particularly useful when writing custom scripts that accept a commit reference. Instead of parsing the commit reference manually, you can let git rev-parse normalize the input for you.

Refs

A ref is an indirect way of referring to a commit. You can think of it as a user-friendly alias for a commit hash. This is Git’s internal mechanism of representing branches and tags.

Refs are stored as normal text files in the .git/refs directory, where .git is usually called .git. To explore the refs in one of your repositories, navigate to .git/refs. You should see the following structure, but it will contain different files depending on what branches, tags, and remotes you have in your repo:

.git/refs/
    heads/
        master
        some-feature
    remotes/
        origin/
            master
    tags/
        v0.9

The heads directory defines all of the local branches in you repository. Each filename matches the name of the corresponding branch, and inside the file you’ll find a commit hash. This commit hash is the location of the tip of the branch. To verify this, try running the following two commands from the root of the Git repository:

# Output the contents of `refs/heads/master` file:
cat .git/refs/heads/master

# Inspect the commit at the tip of the `master` branch:
git log -1 master

The commit hash return by the cat command should match the commit ID displayed by git log.

To change the location of the master branch, all Git has to do is change the contents of the refs/heads/master file. Similarly, creating a new branch is simply a matter of writing a commit hash to a new file. This is part of the reason why Git branches are so lightweight compared to SVN.

The tags directory works the exact same way, but it contains tags instead of branches. The remotes directory lists all remote repositories that you created with git remote as separate subdirectories. Inside each one, you’ll find all the remote branches that have been fetched into your repository.

Specifying Refs

When passing a ref to a Git command, you can either define the full name of the ref, or use a short name and let Git search for a matching ref. You should already be familiar with short names for refs, as this is what you’re using each time you refer to a branch by name.

git show some-feature

The some-feature argument in the above command is actually a short name for the branch. Git resolves this to refs/heads/some-feature before using it. You can also specify the full ref on the command line, like so:

git show refs/heads/some-feature

This avoids any ambiguity regarding the location of the ref. This is necessary, for instance, if you had both a tag and a branch called some-feature. However, if you’re using proper naming conventions, ambiguity between tags and branches shouldn’t generally be a problem.

We’ll see more full ref names in the Refspecs section.

Packed Refs

For large repositories, Git will periodically perform a garbage collection to remove unnecessary objects and compress refs into a single file for more efficient performance. You can force this compression with the garbage collection command:

git gc

This moves all of the individual branch and tag files in the refs folder into a single file called packed-refs located in the top of the .git directory. If you open up this file, you’ll find a mapping of commit hashes to refs:

00f54250cf4e549fdfcafe2cf9a2c90bc3800285 refs/heads/feature
0e25143693cfe9d5c2e83944bbaf6d3c4505eb17 refs/heads/master
bb883e4c91c870b5fed88fd36696e752fb6cf8e6 refs/tags/v0.9

On the outside, normal Git functionality won’t be affected in any way. But, if you’re wondering why your .git/refs folder is empty, this is where the refs went.

Special Refs

In addition to the refs directory, there are a few special refs that reside in the top-level .git directory. They are listed below:

• HEAD – The currently checked-out commit/branch.
• FETCH_HEAD – The most recently fetched branch from a remote repo.
• ORIG_HEAD – A backup reference to HEAD before drastic changes to it.
• MERGE_HEAD – The commit(s) that you’re merging into the current branch with git merge.
• CHERRY_PICK_HEAD – The commit that you’re cherry-picking.

These refs are all created and updated by Git when necessary. For example, The git pull command first runs git fetch, which updates the FETCH_HEAD reference. Then, it runs git merge FETCH_HEAD to finish pulling the fetched branches into the repository. Of course, you can use all of these like any other ref, as I’m sure you’ve done with HEAD.

These files contain different content depending on their type and the state of your repository. The HEAD ref can contain either a symbolic ref, which is simply a reference to another ref instead of a commit hash, or a commit hash. For example, take a look at the contents of HEAD when you’re on the master branch:

git checkout master
cat .git/HEAD

This will output ref: refs/heads/master, which means that HEAD points to the refs/heads/master ref. This is how Git knows that the master branch is currently checked out. If you were to switch to another branch, the contents of HEAD would be updated to reflect the new branch. But, if you were to check out a commit instead of a branch, HEAD would contain a commit hash instead of a symbolic ref. This is how Git knows that it’s in a detached HEAD state.

For the most part, HEAD is the only reference that you’ll be using directly. The others are generally only useful when writing lower-level scripts that need to hook into Git’s internal workings.

Refspecs

A refspec maps a branch in the local repository to a branch in a remote repository. This makes it possible to manage remote branches using local Git commands and to configure some advanced git push and git fetch behavior.

A refspec is specified as [+]<src>:<dst>. The <src> parameter is the source branch in the local repository, and the <dst> parameter is the destination branch in the remote repository. The optional + sign is for forcing the remote repository to perform a non-fast-forward update.

Refspecs can be used with the git push command to give a different name to the remote branch. For example, the following command pushes the master branch to the origin remote repo like an ordinary git push, but it uses qa-master as the name for the branch in the origin repo. This is useful for QA teams that need to push their own branches to a remote repo.

git push origin master:refs/heads/qa-master

You can also use refspecs for deleting remote branches. This is a common situation for feature-branch workflows that push the feature branches to a remote repo (e.g., for backup purposes). The remote feature branches still reside in the remote repo after they are deleted from the local repo, so you get a build-up of dead feature branches as your project progresses. You can delete them by pushing a refspec that has an empty <src> parameter, like so:

git push origin :some-feature

This is very convenient, since you don’t need to log in to your remote repository and manually delete the remote branch. Note that as of Git v1.7.0 you can use the --delete flag instead of the above method. The following will have the same effect as the above command:

git push origin --delete some-feature

By adding a few lines to the Git configuration file, you can use refspecs to alter the behavior of git fetch. By default, git fetch fetches all of the branches in the remote repository. The reason for this is the following section of the .git/config file:

[remote "origin"]
    url = https://git@github.com:mary/example-repo.git
    fetch = +refs/heads/*:refs/remotes/origin/*

The fetch line tells git fetch to download all of the branches from the origin repo. But, some workflows don’t need all of them. For example, many continuous integration workflows only care about the master branch. To fetch only the master branch, change the fetch line to match the following:

[remote "origin"]
    url = https://git@github.com:mary/example-repo.git
    fetch = +refs/heads/master:refs/remotes/origin/master

You can also configure git push in a similar manner. For instance, if you want to always push the master branch to qa-master in the origin remote (as we did above), you would change the config file to:

[remote "origin"]
    url = https://git@github.com:mary/example-repo.git
    fetch = +refs/heads/master:refs/remotes/origin/master
    push = refs/heads/master:refs/heads/qa-master

Refspecs give you complete control over how various Git commands transfer branches between repositories. They let you rename and delete branches from your local repository, fetch/push to branches with different names, and configure git push and git fetch to work with only the branches that you want.

Relative Refs

You can also refer to commits relative to another commit. The ~ character lets you reach parent commits. For example, the following displays the grandparent of HEAD:

git show HEAD~2

But, when working with merge commits, things get a little more complicated. Since merge commits have more than one parent, there is more than one path that you can follow. For 3-way merges, the first parent is from the branch that you were on when you performed the merge, and the second parent is from the branch that you passed to the git merge command.

The ~ character will always follow the first parent of a merge commit. If you want to follow a different parent, you need to specify which one with the ^ character. For example, if HEAD is a merge commit, the following returns the second parent of HEAD.

git show HEAD^2

You can use more than one ^ character to move more than one generation. For instance, this displays the grandparent of HEAD (assuming it’s a merge commit) that rests on the second parent.

git show HEAD^2^1

To clarify how ~ and ^ work, the following figure shows you how to reach any commit from A using relative references. In some cases, there are multiple ways to reach a commit.

Relative refs can be used with the same commands that a normal ref can be used. For example, all of the following commands use a relative reference:

# Only list commits that are parent of the second parent of a merge commit
git log HEAD^2

# Remove the last 3 commits from the current branch
git reset HEAD~3

# Interactively rebase the last 3 commits on the current branch
git rebase -i HEAD~3

The Reflog

The reflog is Git’s safety net. It records almost every change you make in your repository, regardless of whether you committed a snapshot or not. You can think of it is a chronological history of everything you’ve done in your local repo. To view the reflog, run the git reflog command. It should output something that looks like the following:

400e4b7 HEAD@{0}: checkout: moving from master to HEAD~2
0e25143 HEAD@{1}: commit (amend): Integrate some awesome feature into `master`
00f5425 HEAD@{2}: commit (merge): Merge branch ';feature';
ad8621a HEAD@{3}: commit: Finish the feature

This can be translated as follows:

• You just checked out HEAD~2
• Before that you amended a commit message
• Before that you merged the feature branch into master
• Before that you committed a snapshot

The HEAD{<n>} syntax lets you reference commits stored in the reflog. It works a lot like the HEAD~<n> references from the previous section, but the <n> refers to an entry in the reflog instead of the commit history.

You can use this to revert to a state that would otherwise be lost. For example, lets say you just scrapped a new feature with git reset. Your reflog might look something like this:

ad8621a HEAD@{0}: reset: moving to HEAD~3
298eb9f HEAD@{1}: commit: Some other commit message
bbe9012 HEAD@{2}: commit: Continue the feature
9cb79fa HEAD@{3}: commit: Start a new feature

The three commits before the git reset are now dangling, which means that there is no way to reference them—except through the reflog. Now, let’s say you realize that you shouldn’t have thrown away all of your work. All you have to do is check out the HEAD@{1} commit to get back to the state of your repository before you ran git reset.

git checkout HEAD@{1}

This puts you in a detached HEAD state. From here, you can create a new branch and continue working on your feature.

Summary

You should now be quite comfortable referring to commits in a Git repository. We learned how branches and tags were stored as refs in the .git subdirectory, how to read a packed-refs file, how HEAD is represented, how to use refspecs for advanced pushing and fetching, and how to use the relative ~ and ^ operators to traverse a branch hierarchy.

We also took a look at the reflog, which is a way to reference commits that are not available through any other means. This is a great way to recover from those little “Oops, I shouldn’t have done that” situations.

The point of all this was to be able to pick out exactly the commit that you need in any given development scenario. It’s very easy to leverage the skills you learned in this article against your existing Git knowledge, as some of the most common commands accept refs as arguments, including git log, git show, git checkout, git reset, git revert, git rebase, and many others.

Merging vs. Rebasing

The git rebase command has a reputation for being magical Git voodoo that beginners should stay away from, but it can actually make life much easier for a development team when used with care. In this article, we’ll compare git rebase with the related git merge command and identify all of the potential opportunities to incorporate rebasing into the typical Git workflow.

Conceptual Overview

The first thing to understand about git rebase is that it solves the same problem as git merge. Both of these commands are designed to integrate changes from one branch into another branch—they just do it in very different ways.

Consider what happens when you start working on a new feature in a dedicated branch, then another team member updates the master branch with new commits. This results in a forked history, which should be familiar to anyone who has used Git as a collaboration tool.

Now, let’s say that the new commits in master are relevant to the feature that you’re working on. To incorporate the new commits into your feature branch, you have two options: merging or rebasing.

The Merge Option

The easiest option is to merge the master branch into the feature branch using something like the following:

git checkout feature
git merge master

Or, you can condense this to a one-liner:

git merge master feature

This creates a new “merge commit” in the feature branch that ties together the histories of both branches, giving you a branch structure that looks like this:

Merging is nice because it’s a non-destructive operation. The existing branches are not changed in any way. This avoids all of the potential pitfalls of rebasing (discussed below).

On the other hand, this also means that the feature branch will have an extraneous merge commit every time you need to incorporate upstream changes. If master is very active, this can pollute your feature branch’s history quite a bit. While it’s possible to mitigate this issue with advanced git log options, it can make it hard for other developers to understand the history of the project.

The Rebase Option

As an alternative to merging, you can rebase the feature branch onto master branch using the following commands:

git checkout feature
git rebase master

This moves the entire feature branch to begin on the tip of the master branch, effectively incorporating all of the new commits in master. But, instead of using a merge commit, rebasing re-writes the project history by creating brand new commits for each commit in the original branch.

The major benefit of rebasing is that you get a much cleaner project history. First, it eliminates the unnecessary merge commits required by git merge. Second, as you can see in the above diagram, rebasing also results in a perfectly linear project history—you can follow the tip of feature all the way to the beginning of the project without any forks. This makes it easier to navigate your project with commands like git log, git bisect, and gitk.

But, there are two trade-offs for this pristine commit history: safety and traceability. If you don’t follow the Golden Rule of Rebasing, re-writing project history can be potentially catastrophic for your collaboration workflow. And, less importantly, rebasing loses the context provided by a merge commit—you can’t see when upstream changes were incorporated into the feature.

Interactive Rebasing

Interactive rebasing gives you the opportunity to alter commits as they are moved to the new branch. This is even more powerful than an automated rebase, since it offers complete control over the branch’s commit history. Typically, this is used to clean up a messy history before merging a feature branch into master.

To begin an interactive rebasing session, pass the i option to the git rebase command:

git checkout feature
git rebase -i master

This will open a text editor listing all of the commits that are about to be moved:

pick 33d5b7a Message for commit #1
pick 9480b3d Message for commit #2
pick 5c67e61 Message for commit #3

This listing defines exactly what the branch will look like after the rebase is performed. By changing the pick command and/or re-ordering the entries, you can make the branch’s history look like whatever you want. For example, if the 2nd commit fixes a small problem in the 1st commit, you can condense them into a single commit with the fixup command:

pick 33d5b7a Message for commit #1
fixup 9480b3d Message for commit #2
pick 5c67e61 Message for commit #3

When you save and close the file, Git will perform the rebase according to your instructions, resulting in project history that looks like the following:

Eliminating insignificant commits like this makes your feature’s history much easier to understand. This is something that git merge simply cannot do.

The Golden Rule of Rebasing

Once you understand what rebasing is, the most important thing to learn is when not to do it. The golden rule of git rebase is to never use it on public branches.

For example, think about what would happen if you rebased master onto your feature branch:

The rebase moves all of the commits in master onto the tip of feature. The problem is that this only happened in your repository. All of the other developers are still working with the original master. Since rebasing results in brand new commits, Git will think that your master branch’s history has diverged from everybody else’s.

The only way to synchronize the two master branches is to merge them back together, resulting in an extra merge commit and two sets of commits that contain the same changes (the original ones, and the ones from your rebased branch). Needless to say, this is a very confusing situation.

So, before you run git rebase, always ask yourself, “Is anyone else looking at this branch?” If the answer is yes, take your hands off the keyboard and start thinking about a non-destructive way to make your changes (e.g., the git revert command). Otherwise, you’re safe to re-write history as much as you like.

Force-Pushing

If you try to push the rebased master branch back to a remote repository, Git will prevent you from doing so because it conflicts with the remote master branch. But, you can force the push to go through by passing the --force flag, like so:

# Be very careful with this command!
git push --force

This overwrites the remote master branch to match the rebased one from your repository and makes things very confusing for the rest of your team. So, be very careful to use this command only when you know exactly what you’re doing.

One of the only times you should be force-pushing is when you’ve performed a local cleanup after you’ve pushed a private feature branch to a remote repository (e.g., for backup purposes). This is like saying, “Oops, I didn’t really want to push that original version of the feature branch. Take the current one instead.” Again, it’s important that nobody is working off of the commits from the original version of the feature branch.

Workflow Walkthrough

Rebasing can be incorporated into your existing Git workflow as much or as little as your team is comfortable with. In this section, we’ll take a look at the benefits that rebasing can offer at the various stages of a feature’s development.

The first step in any workflow that leverages git rebase is to create a dedicated branch for each feature. This gives you the necessary branch structure to safely utilize rebasing:

Local Cleanup

One of the best ways to incorporate rebasing into your workflow is to clean up local, in-progress features. By periodically performing an interactive rebase, you can make sure each commit in your feature is focused and meaningful. This lets you write your code without worrying about breaking it up into isolated commits—you can fix it up after the fact.

When calling git rebase, you have two options for the new base: The feature’s parent branch (e.g., master), or an earlier commit in your feature. We saw an example of the first option in the Interactive Rebasing section. The latter option is nice when you only need to fix up the last few commits. For example, the following command begins an interactive rebase of only the last 3 commits.

git checkout feature
git rebase -i HEAD~3

By specifying HEAD~3 as the new base, you’re not actually moving the branch—you’re just interactively re-writing the 3 commits that follow it. Note that this will not incorporate upstream changes into the feature branch.

If you want to re-write the entire feature using this method, the git merge-base command can be useful to find the original base of the feature branch. The following returns the commit ID of the original base, which you can then pass to git rebase:

git merge-base feature master

This use of interactive rebasing is a great way to introduce git rebase into your workflow, as it only affects local branches. The only thing other developers will see is your finished product, which should be a clean, easy-to-follow feature branch history.

But again, this only works for private feature branches. If you’re collaborating with other developers via the same feature branch, that branch is public, and you’re not allowed to re-write its history.

There is no git merge alternative for cleaning up local commits with an interactive rebase.

Incorporating Upstream Changes Into a Feature

In the Conceptual Overview section, we saw how a feature branch can incorporate upstream changes from master using either git merge or git rebase. Merging is a safe option that preserves the entire history of your repository, while rebasing creates a linear history by moving your feature branch onto the tip of master.

This use of git rebase is similar to a local cleanup (and can be performed simultaneously), but in the process it incorporates those upstream commits from master.

Keep in mind that it’s perfectly legal to rebase onto a remote branch instead of master. This can happen when collaborating on the same feature with another developer and you need to incorporate their changes into your repository.

For example, if you and another developer named John added commits to the feature branch, your repository might look like the following after fetching the remote feature branch from John’s repository:

You can resolve this fork the exact same way as you integrate upstream changes from master: either merge your local feature with john/feature, or rebase your local feature onto the tip of john/feature.

Note that this rebase doesn’t violate the Golden Rule of Rebasing because only your local feature commits are being moved—everything before that is untouched. This is like saying, “add my changes to what John has already done.” In most circumstances, this is more intuitive than synchronizing with the remote branch via a merge commit.

By default, the git pull command performs a merge, but you can force it to integrate the remote branch with a rebase by passing it the --rebase option.

Reviewing a Feature With a Pull Request

If you use pull requests as part of your code review process, you need to avoid using git rebase after creating the pull request. As soon as you make the pull request, other developers will be looking at your commits, which means that it’s a public branch. Re-writing its history will make it impossible for Git and your teammates to track any follow-up commits added to the feature.

Any changes from other developers need to be incorporated with git merge instead of git rebase.

For this reason, it’s usually a good idea to clean up your code with an interactive rebase before submitting your pull request.

Integrating an Approved Feature

After a feature has been approved by your team, you have the option of rebasing the feature onto the tip of the master branch before using git merge to integrate the feature into the main code base.

This is a similar situation to incorporating upstream changes into a feature branch, but since you’re not allowed to re-write commits in the master branch, you have to eventually use git merge to integrate the feature. However, by performing a rebase before the merge, you’re assured that the merge will be fast-forwarded, resulting in a perfectly linear history. This also gives you the chance to squash any follow-up commits added during a pull request.

If you’re not entirely comfortable with git rebase, you can always perform the rebase in a temporary branch. That way, if you accidentally mess up your feature’s history, you can check out the original branch and try again. For example:

git checkout feature
git checkout -b temporary-branch
git rebase -i master
# [Clean up the history]
git checkout master
git merge temporary-branch

Summary

And that’s all you really need to know to start rebasing your branches. If you would prefer a clean, linear history free of unnecessary merge commits, you should reach for git rebase instead of git merge when integrating changes from another branch.

On the other hand, if you want to preserve the complete history of your project and avoid the risk of re-writing public commits, you can stick with git merge. Either option is perfectly valid, but at least now you have the option of leveraging the benefits of git rebase.

We recently encountered a connection leak with one of our services in production. Here’s what happened, what we did and the lessons learned…

Tests and Monitoring detected the issue (silently)

Our automated tests (which run in production for this service) started failing soon after the incident occurred. We didn’t immediately realise since they just went red on the dashboard and we just carried on about our business. Our monitoring detected the issue immediately and also went red but crucially didn’t email us due to an environment specific config issue.

The alarm is raised

A tester on the team realises that the tests had been failing for some time (some time being 4 hours… eek!) and gets us to start investigating.

Myself and another dev quickly looked into what was going on with our poorly dropwizard app. We could see from the monitoring dashboard that the application’s health check page was in error. Viewing the page showed us that a downstream service (being called over http) was resulting in the following exception…

com.sun.jersey.api.client.ClientHandlerException:
org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool

My gut reaction was that this was a timeout issue (since it usually is) and connections were not being released back to the connection pool after either slow connections or responses. We saved the metrics info from the application and performed the classic java kill -3 to get a thread dump for later reading. We tried checking connectivity from the application’s host to the downstream service with telnet which revealed no issue. We also tried simulating a http request with wget which (after a bit of confusion trying to suppress ssl certificate checking) also revealed no issues. With no more debug ideas immediately coming to mind, restoring service had to be the priority. A restart was performed and everything went green again.

Looking in the code revealed no obvious problems. We had set our timeouts like good boy scouts and just couldn’t figure out what was wrong. It clearly needed more investigation.

We see the same thing happening in our Test environment… Great!

With a lot more time to debug (what seemed like) the exact same issue we came up with the idea of looking at the current connections on the box….

netstat -a

This revealed a lot of connections in the CLOSE_WAIT state.

A simple word count….

netstat -a  | grep CLOSE_WAIT | wc -l

…gave us 1024 which just so happened to be the exact maximum size of our connection pool. The connection pool was exhausted. Restarting the application bought this down to zero and restored service as it did in production.

Now we can monitor our leak

With the command above, we were able to establish how many connections in the pool were in the CLOSE_WAIT state. The closer it gets to 1024, the closer we are to another outage. We then had the great idea of piping this command into mailx and spamming the team with the number every few hours.

Until we fix the connection leak - we need to keep an eye on these emails and restart the application before the bucket over fills (i.e. the connection pool is exhausted).

What caused the leak?

We were using Jersey which in-turn was using HttpComponents4 to send a http request to another service. The service being called was very simple and did not return any response body, just a status code which was all we were interested in. If the status code was 200, the user was logged in, anything else we treat as unauthorised.

The code below shows our mistake….

public boolean isSessionActive(HttpServletRequest httpRequest) {
   ClientResponse response = jerseyClient
      .resource(url)
      .type(APPLICATION_JSON_TYPE)
      .header("Cookie", httpRequest.getHeader("Cookie"))
      .get(ClientResponse.class);
  return response.getStatus() == 200;
}

We were not calling close() on the ClientResponse object which meant that the response’s input stream was not being closed, which meant that the connection was not released back into the pool. This was a silly oversight on our part. However, there’s a twist…

We had run performance tests which in turn executed this method a million times (far more invocations than connections in the pool) without us ever hitting any issues. The question was, if the code was wrong, why was it not failing all the time?

For every bug, there’s a missing test

The task of creating a test which would fail reliably each time due to this issue was 99% of the effort in understanding what was going on. Before we identified the above culprit, a tester on our team, amazingly, found the exact scenario which caused the leased connections to increase. It seemed that each time the service responded with a 403, the number of leased connections would increase by one.

Our component tests mocked the service being called in the above code with a great tool called wiremock which allows you to fully mock a web service by sending http requests to a server under your control. When we examined the number of leased connections after the mocked 403 response, frustratingly it did not increase. There was something different from our mocked 403 response and the actual 403 response.

The issue was, in a deployed environment, we called the webservice through a http load balancer that was configured to give an html error page if the response was 4xx. This scenario created an input stream on the ClientResponse object which needed to be closed.

As soon as we adjusted our wiremock config to return a response body, we observed the same connection leak as in our deployed environment. Now we could write a test that would fail due to the bug. Rather than stop there, we concluded that there was no point in us checking the status after the one test alone as next time it could fail in a different place. We added the following methods and annotations to our test base class:

@Before
public void initialiseLeasedConnectionsCount() throws Exception{
   leasedConnectionCounts = getLeasedConnections();
}

@After
public void checkLeasedConnectionsHasNotIncremented() throws Exception{
   int leasedConnectionsAfter = getLeasedConnections();
   if (leasedConnectionCounts != leasedConnectionsAfter){
      fail("Expected to see leasedConnections stay the same but increased from : " + leasedConnectionCounts + " to: " + leasedConnectionsAfter);
   }
}

If we hit another connection leak, we’ll know the exact scenario and it shouldn’t get anywhere near production this time. Now we have a failing test, lets make it pass by fixing the connection leak….

public boolean isSessionActive(HttpServletRequest httpRequest) {
   ClientResponse response = null;
   try{
      response = jerseyClient
         .resource(url)
         .type(APPLICATION_JSON_TYPE)
         .header("Cookie", httpRequest.getHeader("Cookie"))
         .get(ClientResponse.class);
      return response.getStatus() == SC_OK;
   }
   finally {
      if (response != null){
         response.close();
      }
   }
}

Lessons Learned

Make sure your alerts alert

Until you have proven you are actually getting emails to your inbox, all your wonderfully clever monitoring/alerting may be useless. We lost 4 hours of availability this way!
I think that this is so important that I’m tempted to say it’s worth causing an incident when everyone is paying attention just to prove you have all the bases covered. Netflix do this as routine and at random times too.

A connection pool almost exhausted should fire an alert

If we had received an alert when the connection pool was at around 75% capacity, we would have investigated, our connection leak before it caused an outage. This has also been mentioned by bitly here. I will endeavour to get this in place ASAP.

Learn your tools

Getting a thread dump by sending your java process the SIGQUIT (i.e. kill -3 [java process id]) is fine but it relies on you having access to the box and permissions to run it. In our case (using dropwizard and metrics) all I needed to do was go to this url /metrics/threads on the admin port for exactly the same data formatted in json.

Other Lessons?

If you can spot any other lessons learned from this, feel free to comment.

EDIT: See here for a more detailed explanation of how to test for connection leaks and this time with all the code!

Installing plugins

I recommend using vim-plug to install plugins. It’s simple, works on both Vim and Neovim, and can perform operations asynchronously.

• vim-plug

Syntax

Vim comes with a JavaScript syntax file. It is automatically loaded when you open up a JavaScript file (where the value of filetype is javascript).

By default Vim doesn’t automatically detect filetypes, so if you aren’t using vim-plug, you’ll need to enable it manually by adding this to your vimrc:

filetype plugin indent on

This will enable filetype detection, running filetype specific plugins, and loading filetype specific indentation settings.

You may also need to enable syntax highlighting in your vimrc file if you’re not using vim-plug. To manually enable syntax highlighting, add the following line to your vimrc file:

syntax enable

Plugins that provide better syntax support

The standard JavaScript syntax highlighting is typically adequate, but you may want more features such as ES2015 support, or better distinguishing of keywords.

There are quite a few options:

• pangloss/vim-javascript
  • Includes custom indent settings. These indent settings are also the ones included in the Vim default runtime now, and the one in the plugin may be trailing what comes with Vim (i.e., Vim has a newer version of the same indent file!)
  • Adds special concealing symbols so to make your code pretty at a glance
  • Last updated April 2016 with additional ES2015 support (better arrow function and highlighting among other things) and some regex performance updates.
• sheerun/vim-polyglot
  • This is a plugin that bundles a bunch of language syntax plugins into one. It includes pangloss/vim-javascript at the latest version, as well as some other plugins like mxw/vim-jsx. Worth checking out if you don’t want to maintain syntax plugins on your own, but you should double-check to make sure you aren’t manually adding the bundled plugins outside of the pack (resulting in having the plugin twice).
• jelera/vim-javascript-syntax
  • Does not include custom indent settings (not that you need them…)
  • Updated about once a month according to the GitHub contributors graph
• othree/yajs.vim
  • This is a fork of jelera/vim-javascript-syntax
  • Does not include custom indent settings (again, you might not need one)
  • Updated very often to keep in line with ES specifications
• bigfish/vim-js-context-coloring
  • This is an interesting new method of syntax highlighting. It picks out function scopes from your program by runningg it through a node.js binary that runs a JavaScript parser and assigns a color to the scope. Things within that scope are assigned a color. Because it requires in-depth parsing of your code, it may not color your code when it is incomplete (i.e., the syntax is not yet valid).
  • This syntax plugin can be used in combination with any of the above, and you can toggle it on and off.

With the exception of vim-js-context-coloring, all of these provide ES2015 (ES6) and JSDoc support to varying degrees, which Vim lacks by default. I personally use and recommend pangloss/vim-javascript purely because it has started active development again. I have not experienced any issues since switching to this from othree/yajs.vim.

othree/yajs.vim with es.next.syntax.vim might better suit your needs if you use ES2016 (ES7). The author also writes a few other plugins that can be used in conjunction with it (although they may work with the other syntaxes, too, depending on the syntax groups provided). I’ve used this plugin extensively in place of the default syntax and haven’t had any problems.

Using vim-plug, I recommend installing the plugin like this:

Plug 'othree/yajs.vim', { 'for': 'javascript' }

The additional requirement at the end makes sure the syntax plugin is loaded in a Vim autocommand based on filetype detection (as opposed to relying on Vim’s runtimepath based sourcing mechanism. This way the main Vim syntax plugin will have already run, and the plugin’s syntax will override it.

Plugins that provide better indentation support

Vim’s bundled JavaScript indent may be enough for you, especially if you use a strictly C-style whitespace (Vim’s C-style indent options is even called cindent). Vim, as of August 26, 2016, comes with a modified version of the pangloss/vim-javascript indent rules that does not include the reformatting gq action. This means 99% of people probably won’t need to change their indent file.

Some notable options in this case are:

• pangloss/vim-javascript
  • Using this entire syntax plugin will provide you with a more JavaScript-y indent for things like switch/case and multi-line var declarations.
  • This indent plugin has a side-effect in that it also changes the format expression – that is, if you highlight a block and use gq to reformat it, it will re-indent the code using this plugin as well.
  • The Vim runtime (stuff that comes with Vim) may include a newer version of this indent file that does not have the modified gq.
• gavocanov/vim-js-indent
  • I don’t recommend this one, but I’ve listed it here because you’ll probably find it and have questions. The author no longer uses it, so it should not be considered.
  • This is the indent portion of pangloss/vim-javascript, ripped out into its own plugin.
  • There are modifications to it, diverging it from pangloss/vim-javascript, notably support for the syntax group names in othree/yajs.vim. That helps it pick out keywords and when you are inside comments if you are using othree/yajs.vim.
  • The format expression from the pangloss plugin has been removed.
• itspriddle/vim-javascript-indent
  • This is a git mirror of Ryan Fabella’s indent script
  • It mostly detects closing brackets and parentheses and indents based on what it finds.
• jiangmiao/simple-javascript-indenter
  • An indent plugin with an option for alignment under functions.
• jason0x43/vim-js-indent
  • This is a somewhat new (last updated 2014, as of this writing) indent script that also has some TypeScript support. It is based on an older indent script by Tye Zdrojewski and the default JS-in-HTML indent logic that comes with Vim.
  • The indent logic starts with normal cindent styles and adds special cases for comments, JSDoc, arrays, and switch/case.

Your best bet is to stick with what comes with Vim by default, and only try out the others if your coding style does not match what the rest of the JavaScript community is converging towards. If there’s a quirk, it is probably better to submit an issue with the pangloss/vim-javascript repo.

Related syntaxes

During JavaScript development you may find yourself editing a lot of other filetypes that plain JavaScript.

es.next / ES2016 / ES7 support

othree has a syntax plugin that provides support for planned, but not-yet-final EcmaScript features: othree/es.next.syntax.vim. I personally don’t use it since I currently stick to the ES2015 feature set at most.

• othree/es.next.syntax.vim

JSX support for React

If you write React and use its optional JSX syntax, adding the following plugin will provide you with syntax highlighting for those inline XML-nodes:

Plug 'mxw/vim-jsx'

This plugin requires a JavaScript syntax plugin from above to define certain JavaScript regions. It specifically mentions pangloss’ extension in the docs but actually supports any of them.

• mxw/vim-jsx

JSON

A lot of things use JSON for configuration, so I recommend elzr/vim-json for that. Check out its options, though; I don’t like some of the defaults so I turn them off, but you might want them. Install with:

Plug 'elzr/vim-json'

• elzr/vim-json

JSDoc syntax highlighting

If you document using JSDoc, all of the syntax plugins above support JSDoc highlighting already. There’s a plugin called othree/jsdoc-syntax.vim that pulls that support out of othree/yajs.vim, but it is only for adding JSDoc support to other languages like TypeScript.

• othree/jsdoc-syntax.vim

JSDoc auto-snippets

The plugin heavenshell/vim-jsdoc can automatically insert JSDoc comments for you if your cursor is on a function definition. It’ll check for the function name, arguments, and add the doc-block comment for you.
I use this plugin quite often (I actually have a fork of it with some additions but hopefully they get merged into the upstream).

• heavenshell/vim-jsdoc

jQuery plugins

Files named jquery.*.js are typically jQuery plugins. There’s a specific syntax highlighting plugin for such files: itspriddle/vim-jquery, but it’s pretty old and you’ll have better support combining an up-to-date syntax plugin with the JavaScript libraries plugin in the next section.

• itspriddle/vim-jquery

JavaScript libraries

othree has a syntax plugin, othree/javascript-libraries-syntax.vim, that supports special highlighting of functions and keywords for various libraries such as jQuery, lodash, React, Handlebars, Chai, etc. For an extensive list, see the README at the plugin’s homepage.

I personally do use this plugin.

• othree/javascript-libraries-syntax.vim

JavaScript code completion for Vim

Omni completion

I won’t go into too much detail about this since it isn’t JavaScript specific.

Vim includes basic code completion built in. See this wikia article for information on how to use that. The gist is that the completion system will run a function, the omnifunc, to populate autocompletion pop-up with results.

To use the default completion function, add this to your vimrc:

autocmd FileType javascript setlocal omnifunc=javascriptcomplete#CompleteJS

For even better completion, consider using a plugin like Shougo/neocomplete.vim or Valloric/YouCompleteMe. On Neovim, an option is Shougo/deoplete.nvim.

These plugins will add features like automatically popping up the completion menu, caching of keywords, and integration with other sources of completion than what’s in the current Vim buffer (allowing for multiple omnifuncs).

For portability across my systems without needing a recompile, I use Shougo/neocomplete.vim. Shougo’s plugins and YouCompleteMe both offer roughly the same feature set, though, so whatever might be missing in one can probably be configured into it.

With Shougo/neocomplete.vim, using multiple sources of completion can be done by providing a list of function names like so:

let g:neocomplete#sources#omni#functions.javascript = [
    \   'jspc#omni',
    \   'tern#Complete',
    \ ]

Extended omni-completion

The plugin 1995eaton/vim-better-javascript-completion provides a somewhat up-to-date JavaScript with HTML5 methods (e.g. localStorage and canvas methods).

This plugin creates a new omni-completion function, js#CompleteJS, and replaces your current JS omnifunc with that. That means you’ll have to use a completion plugin or write some VimL yourself if you want to use it in conjunction with another omnifunc like TernJS in the next section.

• 1995eaton/vim-better-javascript-completion

Code-analysis based completion via TernJS

TernJS is kind of like IntelliSense if you’ve ever used Visual Studio, or like the autocompletion for many very robust IDEs. It parses your code and extracts various symbols, like function names, variable names, and values.

The official vim plugin can also show you function signatures (what parameters parameters it expects) and can extract values from related files (e.g. CommonJS require()‘d files) if you configure it to do so (via a .tern-project).

The completion is provided to Vim’s auto-completion engine via an omnifunc, tern#Complete. Again, you’ll have to setup your omnifunc appropriately to use TernJS results instead of the default omni-completion results.

Installing via vim-plug, which can run additional commands before plugin installation, is done like this:

Plug 'marijnh/tern_for_vim', { 'do': 'npm install' }

This will install its npm dependencies for you (Tern runs a node-based analyzer in the background while you’re editing).

I use this plugin with many of its extra features turned off, just keeping the completion.

• marijnh/tern_for_vim

Function parameter completion

othree has a plugin called JavaScript Parameter Complete that detects when you’re inside a function argument and provides some common autocomplete suggestions for it. This is not a feature that TernJS provides, since Tern only adds existing symbols. For example, if you’re writing an event listener, it’ll suggest things like click, and mouseover for you. You can see all the suggestions it provides in its GitHub source. Install the plugin via:

Plug 'othree/jspc.vim'

On load, the jspc.vim plugin automatically detects whatever omnifunc you already have set as your default. It wraps it with the parameter completion, and falls back to your default if you are not in a parameter completion. Because of this you should specify jspc#omni instead of whatever your default completion is (typically javascriptcomplete#CompleteJS).

• othree/jspc.vim

Code navigation

Jumping between CommonJS modules

The plugin moll/vim-node adds keybindings like for jumping to files in your CommonJS require statements.

Plug 'moll/vim-node'

• moll/vim-node

CTags - Symbol based navigation

CTags are lists of all symbols in your projects (function names, variable names, filenames, etc.). Vim provides ctag support by default, with keybindings to jump to declarations and definitions, and there are a slew of plugins (e.g. ludovicchabant/vim-gutentags) that can auto-generate the tags file for you. Using majutsushi/tagbar, Shougo/unite.vim, ctrlpvim/ctrlp.vim, or a bunch of other plugins (and plugins that work with them – plugin-plugins), you can browse through those tags.

Of particular note on the generation side is ramitos/jsctags, which will generate ctags using TernJS.

Personally, I find ctags too annoying to use since the files need to be regenerated to search them (except with majutsushi/tagbar which runs ctags on every open, but only for the current file). Usually just using git grep or the_silver_searcher is adequate, and there are plugins for those, too (out of scope for this article).

• ramitos/jsctags

Linting

While there are actually JSHint, JSLint, eslint, etc. runners for Vim, for your own sanity just use scrooloose/syntastic. It supports a variety of syntax checkers, but you may need to install them first. For example, for eslint support, which is the standard these days, npm install -g eslint first. Refer to syntastic’s wiki page on configuring various JavaScript linters.

Syntastic’s pitfalls are that it is large (it is essentially a linter framework for Vim) and it doesn’t run asynchronously (doesn’t mean it is slow though – depends on the speed of the lint program).

You could alternatively use Vim’s built-in makeprg, which can run any program and output the results to Vim, but you miss out on things like using multiple makeprgs at a time (e.g. JSCS, eslint, and the flow type checker at once) and grouping results. Syntastic actually uses makeprg under the covers, so besides the minimal overhead of configuring some variables it really isn’t any slower.

There’s osyo-manga/vim-watchdogs, which runs linters asynchronously, but the docs are only in Japanese. The plugins Shougo/vimproc.vim and tpope/vim-dispatch can run any tool async, but they aren’t easily configurable as lint-runners. If you follow modern JS design patterns, your JavaScript files should ideally be small modules so running linters asynchronously won’t provide noticeable benefit.

If you’re running Neovim, neomake is an option that’s gaining popularity. It makes full use of Neovim’s asynchronous job support.

• scrooloose/syntastic
• osyo-manga/vim-watchdogs
• neomake

Formatting

Vim has a built-in re-formatter for whitespace. Visually select some text and use the = key to re-indent all of it.

For minified JS, there’s a vim plugin, vim-jsbeautify, that can run your code through jsbeautifier for you. I personally don’t use this and prefer to go to the jsbeautifier website, copying and pasting it there if I need to un-minify something.

There’s also Chiel92/vim-autoformat, which supports formatters for many different languages. Of note is jscs and js-beautifier support.

• vim-jsbeautify
• Chiel92/vim-autoformat

My Vim setup

You can dig through my Vim configuration on GitHub. The plugins I use are all in the main vimrc file, and their configurations are interspersed into plugin/, ftplugin/, and after/*/ to cope with the order in which Vim loads files.

• my Vim configuration on GitHub

It has come to my attention that somebody ‘round these parts has been preaching the gospel of Sublime Text 3. Well, as the resident senior developer (err, old fogey), I feel it’s my duty to provide a counterexample by way of the only TRUE Python development environment you will ever need – and that environment is of course VIM. That’s right. VIM is ubiquitous, fast, and never crashes. And it can do just about anything!

On the down side though, VIM can be a pain to configure, but fear not – This article will show you how to get a powerful VIM environment setup, geared towards wrangling Python day in and day out.

To get the most out of this article you should have at least a basic understanding on how to use VIM and its command modes. If you’re just getting started, check out this resource or this one. Spend some time with it before moving on.

Installing

Since VIM comes pre-installed on a number of *nix systems, let’s first and foremost check to see if it’s installed:

vim --version

If installed, you should see something like:

VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Nov  5 2014 21:00:28)
Compiled by root@apple.com
Normal version without GUI.  Features included (+) or not (-):
-arabic +autocmd -balloon_eval -browse +builtin_terms +byte_offset +cindent
-clientserver -clipboard +cmdline_compl +cmdline_hist +cmdline_info +comments
-conceal +cryptv +cscope +cursorbind +cursorshape +dialog_con +diff +digraphs
-dnd -ebcdic -emacs_tags +eval +ex_extra +extra_search -farsi +file_in_path
+find_in_path +float +folding -footer +fork() -gettext -hangul_input +iconv
+insert_expand +jumplist -keymap -langmap +libcall +linebreak +lispindent
+listcmds +localmap -lua +menu +mksession +modify_fname +mouse -mouseshape
-mouse_dec -mouse_gpm -mouse_jsbterm -mouse_netterm -mouse_sysmouse
+mouse_xterm +multi_byte +multi_lang -mzscheme +netbeans_intg -osfiletype
+path_extra -perl +persistent_undo +postscript +printer -profile +python/dyn
-python3 +quickfix +reltime -rightleft +ruby/dyn +scrollbind +signs
+smartindent -sniff +startuptime +statusline -sun_workshop +syntax +tag_binary
+tag_old_static -tag_any_white -tcl +terminfo +termresponse +textobjects +title
 -toolbar +user_commands +vertsplit +virtualedit +visual +visualextra +viminfo
+vreplace +wildignore +wildmenu +windows +writebackup -X11 -xfontset -xim -xsmp
 -xterm_clipboard -xterm_save
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
      user exrc file: "$HOME/.exrc"
  fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -D_FORTIFY_SOURCE=0 -Iproto -DHAVE_CONFIG_H -arch i386 -arch x86_64 -g -Os -pipe
Linking: gcc -arch i386 -arch x86_64 -o vim -lncurses

At this point, you want to look for two things-

1. The VIM version. This should be > 7.3.
2. Python Support. Under the list of features, make sure you see +python.

If you have both of these, then move right along to VIM Extensions. If not, it’s time to install/upgrade.

OSX

Grab Homebrew, if you don’t already have it, and run:

$ brew update
$ brew install vim

*NIX

For Debian or Ubuntu you can try:

$ sudo apt-get remove vim-tiny
$ sudo apt-get update
$ sudo apt-get install vim

For other flavors of Linux check the docs from your package manager. Here is a link to get you started: Install Vim

Windows

There are many different ways to install VIM on Windows. Start with the official docs.

Verifying the Install

Make sure you have installed VIM > 7.3 with Python support. Again, run vim --version to verify this. If you want to check the specific version of Python used in VIM, run :python import sys; print(sys.version) from within VIM:

2.7.6 (default, Sep  9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)]

This should output your current version of Python. If you get an error, then you don’t have Python support and you either need to reinstall or recompile if you’re building from source.

With VIM installed, let’s look at how to customize VIM for Python development.

VIM Extensions

VIM can do a lot of what developers need right out of the box. However, VIM is also massively extensible and there are some pretty killer extensions that make VIM behave more like a “modern” IDE. So, the very first thing you need is a good extension manager.

Extensions in VIM are often referred to as bundles or plugins.

Vundle

VIM has several extension managers, but the one we strongly recommend is Vundle. Think of it as pip for VIM. It makes installing and updating packages trivial.

Let’s get Vundle installed:

$ git clone https://github.com/gmarik/Vundle.vim.git ~/.vim/bundle/Vundle.vim

This command downloads the Vundle plugin manager and chucks it in your VIM bundles directory. Now you can manage all your extensions from the .vimrc configuration file.

Add the file to your user’s home directory:

$ touch ~/.vimrc

Now set up Vundle in your .vimrc by adding the following to the top of the file:

set nocompatible              " required
filetype off                  " required

" set the runtime path to include Vundle and initialize
set rtp+=~/.vim/bundle/Vundle.vim
call vundle#begin()

" alternatively, pass a path where Vundle should install plugins
"call vundle#begin('~/some/path/here')

" let Vundle manage Vundle, required
Plugin 'gmarik/Vundle.vim'

" Add all your plugins here (note older versions of Vundle used Bundle instead of Plugin)


" All of your Plugins must be added before the following line
call vundle#end()            " required
filetype plugin indent on    " required

That’s it. You’re now set up to use Vundle. Afterward you can add the plugins you want to install, then fire up VIM and run:

:PluginInstall

This command tells Vundle to work its magic – downloading all the plugins and installing/updating them for you.

For Windows users, check out the Windows Installation Instructions.

Let’s make an IDE

We couldn’t possibly list all the VIM features, but let’s look at a quick list of some of the powerful out-of-the-box features perfect for Python development.

Ditch the Mouse

Probably the MOST important feature of VIM is that it doesn’t require a mouse (except for the graphical variants of VIM). At first this may seem like a horrible idea, but after you invest the time – and it does take time – to learn the key combinations, you will speed up your overall workflow!

Split Layouts

Open a file with :sv <filename> and you split the layout vertically (e.g., the new file opens below the current file) or reverse the keys to :vs <filename> and you get a horizontal split (e.g., the new file opens to the right of your current file).

You can nest splits as well, so you can have splits inside of splits, horizontal and vertical, to your heart’s content. As we all know we often need to look at several files at once when developing.

Pro Tip: Make sure to utilize tab completion to find files after typing :sv.

Pro Tip: You can also specify different areas of the screen where the splits should occur by adding the following lines to the .vimrc file:

set splitbelow
set splitright

Pro Tip: Want to move between the splits without using the mouse? Simply add the following to .vimrc and you can jump between splits with just one key combination:

"split navigations
nnoremap <C-J> <C-W><C-J>
nnoremap <C-K> <C-W><C-K>
nnoremap <C-L> <C-W><C-L>
nnoremap <C-H> <C-W><C-H>

Key combos:

• Ctrl-j move to the split below
• Ctrl-k move to the split above
• Ctrl-l move to the split to the right
• Ctrl-h move to the split to the left

In other words, press Ctrl plus the standard VIM movement key to move to a specific pane.

But wait what is the nnoremap thing? – in a nutshell nnoremap remaps one key combination to another; the no part means remap the key in normal mode as opposed to visual mode. Basically nnoremap <C-J> <C-W><C-j> says in normal mode when I hit <C-J> do <C-W><C-j> instead. More info can be found here.

Buffers

While VIM can do tabs, many prefer buffers and splits. You can think of a buffer as a recently opened file. VIM provides easy access to recent buffers, just type :b <buffer name or number> to switch to an open buffer (autocomplete works here as well). You can also use :ls to list all buffers.

Pro Tip: At the end of the :ls output, VIM will prompt with Hit enter to continue. You can instead type :b <buffer number> and pick the buffer immediately while you still have the list displayed (which saves a keystroke and having to remember the buffer number).

Code Folding

Most “modern” IDEs provide a way to collapse (or fold) methods and classes, showing you just the class/method definition lines instead of all the code.

You can enable that in .vimrc with the following lines:

" Enable folding
set foldmethod=indent
set foldlevel=99

This works all right, but you have to type za to fold (and unfold). The space key would be much better. So add this line to your .vimrc file as well:

" Enable folding with the spacebar
nnoremap <space> za

Now you can easily hide portions of your code that you’re not currently working on.

The initial command, set foldmethod=indent, creates folds based upon line indents. This however often creates more folds that you really want. But have no fear! There are several extensions that attempt to rectify that. We recommend SimpylFold. Install it with Vundle by adding the following line to .vimrc

Plugin 'tmhedberg/SimpylFold'

Don’t forget to install the plugin – :PluginInstall.

Pro Tip: Want to see the docstrings for folded code?

let g:SimpylFold_docstring_preview=1

Python Indentation

Of course for code folding to work based on indentations, you want your indents to be correct. Again VIM falls short a bit out of the box because it doesn’t handle auto-indent after a function definition. Two things can be done with indentation:

1. The first is to get indentation to follow PEP8 standards.
2. The second part is to better handle auto-indentation.

PEP8

To add the proper PEP8 indentation, add the following to your .vimrc:

au BufNewFile,BufRead *.py
    \ set tabstop=4
    \ set softtabstop=4
    \ set shiftwidth=4
    \ set textwidth=79
    \ set expandtab
    \ set autoindent
    \ set fileformat=unix

This will give you the standard four spaces when you hit tab, ensure your line length doesn’t go beyond 80 characters, and store the file in a unix format so you don’t get a bunch of conversion issues when checking into GitHub and/or sharing with other users.

And for full stack development you can use another au command for each filetype:

au BufNewFile,BufRead *.js, *.html, *.css
    \ set tabstop=2
    \ set softtabstop=2
    \ set shiftwidth=2

This way you can have different settings for different filetypes. There is also a plugin called ftypes which will allow you to have a separate file for each filetype you want to maintain settings for, so use that if you see fit.

Auto-Indentation

Autoindent will help but in some cases (like when a function signature spans multiple lines), it doesn’t always do what you want, especially when it comes to conforming to PEP8 standards. To fix that, we can use the indentpython.vim extension:

Plugin 'vim-scripts/indentpython.vim'

Flagging Unnecessary Whitespace

We also want to avoid extraneous whitespace. We can have VIM flag that for us so that it’s easy to spot – and then remove.

au BufRead,BufNewFile *.py,*.pyw,*.c,*.h match BadWhitespace /\s\+$/

This will mark extra whitespace as bad, and probably color it red.

UTF8 Support

For the most part, you should be using UTF8 when working with Python, especially if you’re working with Python 3. Make sure VIM knows that with the following line:

set encoding=utf-8

Auto-complete

The best plugin for Python auto-complete is YouCompleteMe. Again, use Vundle to install:

Bundle 'Valloric/YouCompleteMe'

Under the hood YouCompleteMe uses a few different auto-completers (including Jedi for Python), and it needs some C libraries to be installed for it to work correctly. The docs have very good installation instructions so I won’t repeat them here, but be sure you follow them.

It works out of the box pretty well, but let’s add a few customizations:

let g:ycm_autoclose_preview_window_after_completion=1
map <leader>g  :YcmCompleter GoToDefinitionElseDeclaration<CR>

The former line ensures that the autocomplete window goes away when you’re done with it, and the latter defines a shortcut for goto definition.

My leader key is mapped to space, so space-g will goto definition of whatever I’m currently on. Helpful when exploring new code.

Virtualenv Support

One issue with the goto definition above is that VIM by default doesn’t know anything about virtualenv, so you have to make VIM and YouCompleteMe aware of your virtualenv by adding the following lines of code to .vimrc:

"python with virtualenv support
py << EOF
import os
import sys
if 'VIRTUAL_ENV' in os.environ:
  project_base_dir = os.environ['VIRTUAL_ENV']
  activate_this = os.path.join(project_base_dir, 'bin/activate_this.py')
  execfile(activate_this, dict(__file__=activate_this))
EOF

This determines if you are running inside a virtualenv, and then switches to that specific virtualenv and sets up your system path so that YouCompleteMe will find the appropriate site packages.

Syntax Checking/Highlighting

You can have VIM check your syntax on each save with the syntastic extension:

Plugin 'scrooloose/syntastic'

Also add PEP8 checking with this nifty little plugin:

Plugin 'nvie/vim-flake8'

Finally, make your code look pretty:

let python_highlight_all=1
syntax on

Color Schemes

Color schemes work in conjunction with the basic color scheme that you are using. Check out solarized for GUI mode, and Zenburn for terminal mode:

Plugin 'jnurmine/Zenburn'
Plugin 'altercation/vim-colors-solarized'

Then just add a bit of logic to define which scheme to use based upon the VIM mode:

if has('gui_running')
  set background=dark
  colorscheme solarized
else
  colorscheme zenburn
endif

Solarized also ships with a dark and light theme. To make switching between them very easy (by pressing F5) add:

call togglebg#map("<F5>")

File Browsing

If you want a proper file tree then NERDTree is the way to go.

Plugin 'scrooloose/nerdtree'

And if you want to use tabs, utilize vim-nerdtree-tabs:

Plugin 'jistr/vim-nerdtree-tabs'

Want to hide .pyc files? Then add the following line:

let NERDTreeIgnore=['\.pyc$', '\~$'] "ignore files in NERDTree

Super Searching

Want to search for basically anything from VIM? Check out ctrlP:

Plugin 'kien/ctrlp.vim'

As expected, press Ctrl-P to enable the search and then just start typing. If your search matches anything close to the file you’re looking for, it will find it. Oh – and it’s not just files; it will find tags as well! For more, check out this YouTube video.

Line Numbering

Turn on line numbers on the side of the screen with:

set nu

Git Integration

Want to perform basic git commands without leaving the comfort of VIM? Then vim-fugitive is the way to go:

Plugin 'tpope/vim-fugitive'

See it in action on VIMcasts.

Powerline

Powerline is a status bar that displays things like the current virtualenv, git branch, files being edited, and much more.

It’s written in Python, and it supports a number of other environments like zsh, bash, tmux, and IPython.

Plugin 'Lokaltog/powerline', {'rtp': 'powerline/bindings/vim/'}

Take a look at the official docs for all the configuration options.

System clipboard

Vim usually has its own clipboard and ignores the system keyboards, but sometimes you might want to cut, copy, and/or paste to/from other applications outside of VIM. On OSX, you can access your system clipboard with this line:

set clipboard=unnamed

VIM in the Shell

And finally, once you’ve mastered VIM and its keyboard shortcuts, you’ll often find yourself getting annoyed with the lack of those same shortcuts in the shell. Fear not, most shells have a VI mode. To turn it on for your shell, add the following line to ~/.inputrc:

set editing-mode vi

Now you will not only be able to use VIM key combos in the shell, but also in the Python interpreter and any other tool that uses GNU Readline (i.e., most database shells). Now you have VIM everywhere!

Conclusion

That’s more or less it (for Python development, at least). There are a ton of other extensions that you can use, as well as alternatives to everything detailed in this post. What are some of your favorite extensions? How have you configured VIM to match your personality?

Here is a link to my current VIM config. Got one of your own? Please share!

Thanks for reading!

Resources

1. VIM Tutor comes with VIM, so once VIM is install just type vimtutor from the command line and the program will teach you how to use VIM by, well, using VIM.
2. VIMcasts are advanced tutorial videos describing how to use many of VIM’s features.
3. Official VIM docs
4. Open Vim
5. Learn Vimscript the Hard Way is perfect for learning vimscript.

Reference: 使用 Travis CI 自动更新 GitHub Pages

A sample configuration for my personal blog hosted on github pages would be,

language: node_js

node_js:
  - "6"

env:
  global:
    - GH_REF: github.com/athrunsun/blog.git
    - secure: "xxxxxx"

branches:
  only:
  - master

# S: Build Lifecycle
install:
  - npm install

before_script:
  - git config --global user.name "xxx"
  - git config --global user.email "xxx@xxx.com"

script:
  - hexo clean
  - hexo generate

after_script:
  - cd public
  - git init
  - git add .
  - git commit -m "Update blog $(date +%Y-%m-%d,%H:%M:%S)"
  - git push -f -q "https://${GH_TOKEN}@${GH_REF}" master:gh-pages
# E: Build LifeCycle

But this requires us to install travis command line client, encrypt our github API token and put the result in env["global"]["secure"].

NOTE that we’re using a quite push with -q option so that we won’t leak our github API token. Also you can remove log permanently from Travis CI’s UI once you see unintentional token leakages.

Solution #2

Reference: How to publish to Github Pages from Travis CI?

A simpler solution (global environment variable GITHUB_API_TOKEN must be set beforehand in Travis CI’s UI),

language: node_js

node_js:
  - "6"

branches:
  only:
  - master

install:
  - npm install

script:
  - hexo clean
  - hexo generate

after_success:
  - cd public
  - git init
  - git add .
  - git -c user.name='travis' -c user.email='travis' commit -m "Update blog $(date +%Y-%m-%d,%H:%M:%S)"
  - git push -f -q "https://athrunsun:${GITHUB_API_TOKEN}@github.com/athrunsun/blog.git" master:gh-pages &> /dev/null

Here &> /dev/null means to discard both standard output and error.

A SOCKS server is a general purpose proxy server that establishes a TCP connection to another server on behalf of a client, then routes all the traffic back and forth between the client and the server. It works for any kind of network protocol on any port. SOCKS Version 5 adds additional support for security and UDP.

The SOCKS server does not interpret the network traffic between client and server in any way, and is often used because clients are behind a firewall and are not permitted to establish TCP connections to servers outside the firewall unless they do it through the SOCKS server.

Most web browsers for example can be configured to talk to a web server via a SOCKS server. Because the client must first make a connection to the SOCKS server and tell it the host it wants to connect to, the client must be “SOCKS enabled.”

On Windows, it is possible to “shim” the TCP stack so that all client software is SOCKS enabled. A free SOCKS shim is available from Hummingbird at http://www.hummingbird.com/products/nc/socks/index.html.

An HTTP proxy is similar, and may be used for the same purpose when clients are behind a firewall and are prevented from making outgoing TCP connections to servers outside the firewall.

However, unlike the SOCKS server, an HTTP proxy does understand and interpret the network traffic that passes between the client and downstream server, namely the HTTP protocol.

Because of this the HTTP proxy can ONLY be used to handle HTTP traffic, but it can be very smart about how it does it. In particular, it can recognize often repeated requests and cache the replies to improve performance.

Many ISPs use HTTP proxies regardless of how the browser is configured because they simply route all traffic on port 80 through the proxy server.

Intro

In China, many “ISP” sucks. Their DNS servers often return incorrect ip address results, is known as DNS poisoning! DNS poisoning is a common and simple way to stop people reaching correct web pages.

Here is a solution to get the correct DNS queries results.

Dependent tools

• Server
  • A VPS server that can access famous public DNS servers correctly, eg. 8.8.8.8 (google dns) or 208.67.222.222 (opendns).
  • SSH server running on that VPS. (Please google: how to setup ssh server)
  • socat (Socket Cat). (Please google: how to setup or install socat)
  • dnsmasq (Optional, for caching).
• Local
  • SSH client
  • socat (Socket Cat)
  • dnsmasq (Optional, for caching).

ssh, socat, dnsmasq are open source softwares which can be found and installed easily.

Samples and Steps

• Server
  • Setup a DNS caching server using dnsmasq. (Optional)
    • install dnsmasq
    • configure example using google dns and opendns servers. please check out: Setup a DNS cache server using dnsmasq
    • start dnsmasq
  • If no local dns server, just use a public dns server instead, eg. 8.8.8.8:53
  • Forwarding UDP to TCP by socat (listen on port: 15353)
    • install socat
    • start socat:
      • if use a public dns server, eg. 8.8.8.8:53
socat tcp4-listen:15353,reuseaddr,fork,bind=127.0.0.1 UDP:8.8.8.8:53
      • if use local dns caching server: 127.0.0.1:5353
socat tcp4-listen:15353,reuseaddr,fork,bind=127.0.0.1 UDP:127.0.0.1:53
  • You can check the forwarding dns server using command line:
    dig +tcp google.com @127.0.0.1 -p 15353
• Local
  • Setup SSH tunnel
    ssh -N -L 15353:localhost:15353 username@vps.ip
  • Forwarding TCP to UDP by socat
    • if no local dns caching server, you can forward to port 53
socat udp-recvfrom:53,reuseaddr,bind=127.0.0.1,fork tcp:127.0.0.1:15353
    • of cause can forward to any port that can be used.
      socat udp-recvfrom:15353,reuseaddr,bind=127.0.0.1,fork tcp:127.0.0.1:15353
  • Setup local dns caching server (Optional but recommend). See the server instruction above.

OK!

Oh not yet!

ssh (tunnel) is not always working well! WTF!

DNS

DNS，Domain Name System或者Domain Name Service（域名系统或者域名服务）。域名系统为Internet上的主机分配域名地址和IP地址。由于网络中的计算机都必须有个IP地址，来识别,互相之间才能通信,但让我们记住一大串的IP地址来访问网站显然是不可能的,所以用户使用域名地址，而DNS系统的功能就是自动把域名地址翻译为IP地址。域名服务是运行域名系统的Internet工具。执行域名服务的服务器称之为DNS服务器，通过DNS服务器来应答域名服务的查询。

1. DNS就是域名服务器，他的任务就是确定域名的解析，比如A记录MX记录等等。
2. 任何域名都至少有一个DNS，一般是2个。但为什么要2个以上呢？因为DNS可以轮回处理，这样第一个解析失败可以找第二个。这样只要有一个DNS解析正常，就不会影响域名的正常使用。
3. 如何确定域名的DNS。很简单到http://www.internic.net/whois.html输入你要查询的域名就可以看到了。这个是国际域名管理中心。唯一的权威。只要这里能查到某个域名，就表示域名是生效的。它说你什么时候到期，就是什么时候到期。
4. 有效的DNS表示当前正在起作用的DNS服务器是谁，比如查询结果是NS.XINNETDNS.COM、NS.XINNET.CN(新网信海)就表示当前域名是由NS.XINNETDNS.COM、NS.XINNET.CN(新网信海)负责解析。其他未显示的DNS的设置，都是无效的。
5. DNS是可以修改的。修改以后需要24-72小时以后，全世界范围的所有DNS服务器才能刷新过来。internic的信息一般在24小时以后可以看到。另外，修改的过程，并不表示域名会停止解析，只要你在2边都做好了解析。如果生效了就是新的DNS在起作用。如果没生效。就是旧的DNS在起作用。要么生效，要么不生效。不存在2个都不起作用的时间。所以域名解析，不会中断。前提是两边都做了解析。

A记录

A = Address，记录是用来指定主机名（或域名）对应的IP地址记录。用户可以将该域名下的网站服务器指向到自己的web server上。同时也可以设置您域名的子域名。通俗来说A记录就是服务器的IP,域名绑定A记录就是告诉DNS,当你输入域名的时候给你引导向设置在DNS的A记录所对应的服务器

子域名

子域名道理等同二级域名，不过比二级域名更加延伸，比如我们继续扩展该域名的主机名，设置主机名为bbs.at,那么就可以建立一个三级域名：bbs.at.abc.com，当然也可以建立四级域名bbs.at.go.abc.com，五级域名bbs.at.go.home.abc.com……，依次类推，可以建立无限级别的域名，我们统称这些域名为顶级域名abc.com的子域名。

CNAME别名指向记录

CNAME = Canonical Name，通常称别名指向。在这里，您可以定义一个主机别名，比如设置ftp.***.com，用来指向一个主机www.***.com,那么以后就可以用FTP.***.com来代替访问www.***.com了。

MX记录

MX记录也叫做邮件路由记录，用户可以将该域名下的邮件服务器指向到自己的mail server上，然后即可自行操控所有的邮箱设置。您只需在线填写您服务器的IP地址，即可将您域名下的邮件全部转到您自己设定相应的邮件服务器上。简单的说，通过操作MX记录，您才可以得到以您域名结尾的邮局。

TXT记录

TXT记录，一般指为某个主机名或域名设置的说明，如：

admin IN TXT "管理员, 电话： 13901234567"
mail IN TXT "邮件主机, 存放在xxx , 管理人：AAA"
Jim IN TXT "contact: abc@mailserver.com"

也就是您可以设置TXT，以便使别人联系到您。

SRV记录

SRV记录：一般是为Microsoft的活动目录设置时的应用。DNS可以独立于活动目录，但是活动目录必须有DNS的帮助才能工作。为了活动目录能够正常的工作，DNS服务器必须支持服务定位（SRV）资源记录，资源记录把服务名字映射为提供服务的服务器名字。活动目录客户和域控制器使用SRV资源记录决定域控制器的IP地址。（此技术细节请参考相应网站）

TTL值

TTL值全称是“生存时间”(Time To Live)，简单的说它表示DNS记录在DNS服务器上缓存时间。要理解TTL值，请先看下面的一个例子：

假设，有这样一个域名myhost.abc.com（其实，这就是一条DNS记录，通常表示在abc.com域中有一台名为myhost的主机）对应IP地址为1.1.1.1，它的TTL为10分钟。这个域名或称这条记录存储在一台名为dns.abc.com的DNS服务器上。

现在有一个用户在浏览器中键入一下地址（又称URL）：http://myhost.abc.com/这时会发生什么呢？

该访问者指定的DNS服务器（或是他的ISP,互联网服务商, 动态分配给他的）8.8.8.8就会试图为他解释myhost.abc.com，当然8.8.8.8这台DNS服务器由于没有包含myhost.abc.com这条信息，因此无法立即解析，但是通过全球DNS的递归查询后，最终定位到dns.abc.com这台DNS服务器，dns.abc.com这台DNS服务器将myhost.abc.com对应的IP地址1.1.1.1告诉8.8.8.8这台DNS服务器，然有再由8.8.8.8告诉用户结果。8.8.8.8为了以后加快对myhost.abc.com这条记录的解析，就将刚才的1.1.1.1结果保留一段时间，这就是TTL时间，在这段时间内如果用户又有对myhost.abc.com这条记录的解析请求，它就直接告诉用户1.1.1.1，当TTL到期则又会重复上面的过程

泛域名与泛解析

泛域名是指在一个域名根下，以*.Domain.com的形式表示这个域名根所有未建立的子域名。

泛解析是把*.Domain.com的A 记录解析到某个IP地址上,然后别人通过任意的前缀.domain.com访问都能访问到你解析的站点上。例：

域名根name.com只建立了www.name.com和name.com这两个域名记录，那么ftp.name.com、mail.name.com、bbs.name.com等域名记录是不存在的，但为域名根name.com建立泛域名记录 *.name.com后，*.name.com就涵盖了ftp.name.com、mail.name.com、bbs.name.com 等所有不存在的子域名记录。

您可以定义*.name.com指向一个IP，那么当访问者无论是输入ftp.name.com、mail.name.com或 bbs.name.com的时候，访问者都将直接访问您定义*.name.com的那个IP。

域名绑定

什么是域名绑定：域名绑定是指，域名绑定是指域名和主机(即某个服务器)的空间进行关联绑定,其实就是在虚拟服务器上设置或者WEB服务器上设置，使一个域名被引导向服务器上的某一特定空间（某一个特定站点），访问者访问你的域名的时候就会打开你存放在该空间上的网页，简单来说其实就是把域名解析到服务器IP,然后在服务器上设置该域名有权限访问的过程。一般虚拟主机有控制面板给进行绑定域名，如果没有面板就需要自己在服务器上的IIS里面设置

域名转向

域名转向我们又称为域名(URL)指向或域名转发，当用户地址栏中输入您的域名时，将会自动跳转到您所指定的另一个网络地址（URL）。

假设abc.com是您注册的域名，则通过URL转发服务可以实现当用户访问http://www.abc.com/时，自动转向访问另外一个URL，如:我的空间不支持绑定到目录功能,所以为了方便访问者,我就设置:http://bbs.bnxb.com来访问,这个是设置了通过URL转发服务转发到http://www.bnxb.com/bbs上面来实现的，这样您就可以轻松实现多个域名指向一个网站或网站子目录；另外，通过URL转发服务，可以方便的实现将您的中文域名，设置自动转发到您的英文域名主站点。域名转发的另外一个好处就是可以把长域名压缩,用短域名的转向代替,让你的访客不需输入那么长的域名。

IIS连接数

IIS连接数指并发连接数，什么意思呢？

要分几种情况：（以100M空间50人在线为例）

1. 用户单点下载你的文件，结束后正常断开，这些连接是按照瞬间计算的，就是说你50人的网站瞬间可以接受同时50个点下载。
2. 用户打开你的页面，就算停留在该页面，这时候网页已经加载完了没有对服务器再发出任何请求，那么在用户完全打开这个页面的时间内算一个在线，在页面完全打开后IIS连接数就释放。
3. 上面B的情况用户继续打开同一个网站的其他页面，那么在线人数按照用户最后一次点击（发出请求）计算，在这个时间内不管用户怎么点击（包括新窗口打开）都还是一人在线。
4. 当你的页面内存在框架(Iframe)，那么每多一个框架就要多一倍的在线！因为这相当于用户同一时间向服务器请求了多个页面。
5. 当用户打开页面然后正常关闭浏览器，用户的在线人数也会马上清除。

The purpose of any version control system is to record changes to your code. This gives you the power to go back into your project history to see who contributed what, figure out where bugs were introduced, and revert problematic changes. But, having all of this history available is useless if you don’t know how to navigate it. That’s where the git log command comes in.

By now, you should already know the basic git log command for displaying commits. But, you can alter this output by passing many different parameters to git log.

The advanced features of git log can be split into two categories: formatting how each commit is displayed, and filtering which commits are included in the output. Together, these two skills give you the power to go back into your project and find any information that you could possibly need.

Formatting Log Output

First, this article will take a look at the many ways in which git log’s output can be formatted. Most of these come in the form of flags that let you request more or less information from git log.

If you don’t like the default git log format, you can use git config’s aliasing functionality to create a shortcut for any of the formatting options discussed below. Please see in The git config Command for how to set up an alias.

Oneline

The --oneline flag condenses each commit to a single line. By default, it displays only the commit ID and the first line of the commit message. Your typical git log --oneline output will look something like this:

0e25143 Merge branch 'feature'
ad8621a Fix a bug in the feature
16b36c6 Add a new feature
23ad9ad Add the initial code base

This is very useful for getting a high-level overview of your project.

Decorating

Many times it’s useful to know which branch or tag each commit is associated with. The --decorate flag makes git log display all of the references (e.g., branches, tags, etc) that point to each commit.

This can be combined with other configuration options. For example, running git log --oneline --decorate will format the commit history like so:

0e25143 (HEAD, master) Merge branch 'feature'
ad8621a (feature) Fix a bug in the feature
16b36c6 Add a new feature
23ad9ad (tag: v0.9) Add the initial code base

This lets you know that the top commit is also checked out (denoted by HEAD) and that it is also the tip of the master branch. The second commit has another branch pointing to it called feature, and finally the 4th commit is tagged as v0.9.

Branches, tags, HEAD, and the commit history are almost all of the information contained in your Git repository, so this gives you a more complete view of the logical structure of your repository.

Diffs

The git log command includes many options for displaying diffs with each commit. Two of the most common options are --stat and -p.

The --stat option displays the number of insertions and deletions to each file altered by each commit (note that modifying a line is represented as 1 insertion and 1 deletion). This is useful when you want a brief summary of the changes introduced by each commit. For example, the following commit added 67 lines to the hello.py file and removed 38 lines:

commit f2a238924e89ca1d4947662928218a06d39068c3
Author: John <john@example.com>
Date:   Fri Jun 25 17:30:28 2014 -0500

    Add a new feature

 hello.py | 105 ++++++++++++++++++++++++-----------------
 1 file changed, 67 insertion(+), 38 deletions(-)

The amount of + and - signs next to the file name show the relative number of changes to each file altered by the commit. This gives you an idea of where the changes for each commit can be found.

If you want to see the actual changes introduced by each commit, you can pass the -p option to git log. This outputs the entire patch representing that commit:

commit 16b36c697eb2d24302f89aa22d9170dfe609855b
Author: Mary <mary@example.com>
Date:   Fri Jun 25 17:31:57 2014 -0500

    Fix a bug in the feature

diff --git a/hello.py b/hello.py
index 18ca709..c673b40 100644
--- a/hello.py
+++ b/hello.py
@@ -13,14 +13,14 @@ B
-print("Hello, World!")
+print("Hello, Git!")

For commits with a lot of changes, the resulting output can become quite long and unwieldy. More often than not, if you’re displaying a full patch, you’re probably searching for a specific change. For this, you want to use the pickaxe option.

The Shortlog

The git shortlog command is a special version of git log intended for creating release announcements. It groups each commit by author and displays the first line of each commit message. This is an easy way to see who’s been working on what.

For example, if two developers have contributed 5 commits to a project, the git shortlog output might look like the following:

Mary (2):
      Fix a bug in the feature
      Fix a serious security hole in our framework

John (3):
      Add the initial code base
      Add a new feature
      Merge branch 'feature'

By default, git shortlog sorts the output by author name, but you can also pass the -n option to sort by the number of commits per author.

Graphs

The --graph option draws an ASCII graph representing the branch structure of the commit history. This is commonly used in conjunction with the --oneline and --decorate commands to make it easier to see which commit belongs to which branch:

git log --graph --oneline --decorate

For a simple repository with just 2 branches, this will produce the following:

*   0e25143 (HEAD, master) Merge branch 'feature'
|\
| * 16b36c6 Fix a bug in the new feature
| * 23ad9ad Start a new feature
* | ad8621a Fix a critical security issue
|/
* 400e4b7 Fix typos in the documentation
* 160e224 Add the initial code base

The asterisk shows which branch the commit was on, so the above graph tells us that the 23ad9ad and 16b36c6 commits are on a topic branch and the rest are on the master branch.

While this is a nice option for simple repositories, you’re probably better off with a more full-featured visualization tool like gitk or SourceTree for projects that are heavily branched.

Custom Formatting

For all of your other git log formatting needs, you can use the --pretty=format:"<string>" option. This lets you display each commit however you want using printf-style placeholders.

For example, the %cn, %h and %cd characters in the following command are replaced with the committer name, abbreviated commit hash, and the committer date, respectively.

git log --pretty=format:"%cn committed %h on %cd"

This results in the following format for each commit:

John committed 400e4b7 on Fri Jun 24 12:30:04 2014 -0500
John committed 89ab2cf on Thu Jun 23 17:09:42 2014 -0500
Mary committed 180e223 on Wed Jun 22 17:21:19 2014 -0500
John committed f12ca28 on Wed Jun 22 13:50:31 2014 -0500

The complete list of placeholders can be found in the Pretty Formats section of the git log manual page.

Aside from letting you view only the information that you’re interested in, the --pretty=format:"<string>" option is particularly useful when you’re trying to pipe git log output into another command.

Filtering the Commit History

Formatting how each commit gets displayed is only half the battle of learning git log. The other half is understanding how to navigate the commit history. The rest of this article introduces some of the advanced ways to pick out specific commits in your project history using git log. All of these can be combined with any of the formatting options discussed above.

By Amount

The most basic filtering option for git log is to limit the number of commits that are displayed. When you’re only interested in the last few commits, this saves you the trouble of viewing all the commits in a pager.

You can limit git log’s output by including the -<n> option. For example, the following command will display only the 3 most recent commits.

git log -3

By Date

If you’re looking for a commit from a specific time frame, you can use the --after or --before flags for filtering commits by date. These both accept a variety of date formats as a parameter. For example, the following command only shows commits that were created after July 1st, 2014 (inclusive):

git log --after="2014-7-1"

You can also pass in relative references like "1 week ago" and "yesterday":

get log --after="yesterday"

To search for a commits that were created between two dates, you can provide both a --before and --after date. For instance, to display all the commits added between July 1st, 2014 and July 4th, 2014, you would use the following:

git log --after="2014-7-1" --before="2014-7-4"

Note that the --since and --until flags are synonymous with --after and --before, respectively.

By Author

When you’re only looking for commits created by a particular user, use the --author flag. This accepts a regular expression, and returns all commits whose author matches that pattern. If you know exactly who you’re looking for, you can use a plain old string instead of a regular expression:

git log --author="John"

This displays all commits whose author includes the name John. The author name doesn’t need to be an exact match — it just needs to contain the specified phrase.

You can also use regular expressions to create more complex searches. For example, the following command searches for commits by either Mary or John.

git log --author="John\|Mary"

Note that the author’s email is also included with the author’s name, so you can use this option to search by email, too.

If your workflow separates committers from authors, the --committer flag operates in the same fashion.

By Message

To filter commits by their commit message, use the --grep flag. This works just like the --author flag discussed above, but it matches against the commit message instead of the author.

For example, if your team includes relevant issue numbers in each commit message, you can use something like the following to pull out all of the commits related to that issue:

git log --grep="JRA-224:"

You can also pass in the -i parameter to git log to make it ignore case differences while pattern matching.

By File

Many times, you’re only interested in changes that happened to a particular file. To show the history related to a file, all you have to do is pass in the file path. For example, the following returns all commits that affected either the foo.py or the bar.py file:

git log -- foo.py bar.py

The -- parameter is used to tell git log that subsequent arguments are file paths and not branch names. If there’s no chance of mixing it up with a branch, you can omit the --.

By Content

It’s also possible to search for commits that introduce or remove a particular line of source code. This is called a pickaxe, and it takes the form of -S"<string>". For example, if you want to know when the string Hello, World! was added to any file in the project, you would use the following command:

git log -S"Hello, World!"

If you want to search using a regular expression instead of a string, you can use the -G"<regex>" flag instead.

This is a very powerful debugging tool, as it lets you locate all of the commits that affect a particular line of code. It can even show you when a line was copied or moved to another file.

By Range

You can pass a range of commits to git log to show only the commits contained in that range. The range is specified in the following format, where <since> and <until> are commit references:

git log <since>..<until>

This command is particularly useful when you use branch references as the parameters. It’s a simple way to show the differences between 2 branches. Consider the following command:

git log master..feature

The master..feature range contains all of the commits that are in the feature branch, but aren’t in the master branch. In other words, this is how far feature has progressed since it forked off of master. You can visualize this as follows:

Note that if you switch the order of the range (feature..master), you will get all of the commits in master, but not in feature. If git log outputs commits for both versions, this tells you that your history has diverged.

Filtering Merge Commits

By default, git log includes merge commits in its output. But, if your team has an always-merge policy (that is, you merge upstream changes into topic branches instead of rebasing the topic branch onto the upstream branch), you’ll have a lot of extraneous merge commits in your project history.

You can prevent git log from displaying these merge commits by passing the --no-merges flag:

git log --no-merges

On the other hand, if you’re only interested in the merge commits, you can use the --merges flag:

git log --merges

This returns all commits that have at least two parents.

Summary

You should now be fairly comfortable using git log’s advanced parameters to format its output and select which commits you want to display. This gives you the power to pull out exactly what you need from your project history.

These new skills are an important part of your Git toolkit, but remember that git log is often used in conjunction other Git commands. Once you’ve found the commit you’re looking for, you typically pass it off to git checkout, git revert, or some other tool for manipulating your commit history. So, be sure to keep on learning about Git’s advanced features.

The git reset, git checkout, and git revert command are some of the most useful tools in your Git toolbox. They all let you undo some kind of change in your repository, and the first two commands can be used to manipulate either commits or individual files.

Because they’re so similar, it’s very easy to mix up which command should be used in any given development scenario. In this article, we’ll compare the most common configurations of git reset, git checkout, and git revert. Hopefully, you’ll walk away with the confidence to navigate your repository using any of these commands.

It helps to think about each command in terms of their effect on the three main components of a Git repository: the working directory, the staged snapshot, and the commit history. Keep these components in mind as you read through this article.

Commit-level Operation

The parameters that you pass to git reset and git checkout determine their scope. When you don’t include a file path as a parameter, they operate on whole commits. That’s what we’ll be exploring in this section. Note that git revert has no file-level counterpart.

Reset

On the commit-level, resetting is a way to move the tip of a branch to a different commit. This can be used to remove commits from the current branch. For example, the following command moves the hotfix branch backwards by two commits.

git checkout hotfix
git reset HEAD~2

The two commits that were on the end of hotfix are now dangling commits, which means they will be deleted the next time Git performs a garbage collection. In other words, you’re saying that you want to throw away these commits. This can be visualized as the following:

This usage of git reset is a simple way to undo changes that haven’t been shared with anyone else. It’s your go-to command when you’ve started working on a feature and find yourself thinking, “Oh crap, what am I doing? I should just start over.”

In addition to moving the current branch, you can also get git reset to alter the staged snapshot and/or the working directory by passing it one of the following flags:

• --soft – The staged snapshot and working directory are not altered in any way.
• --mixed – The staged snapshot is updated to match the specified commit, but the working directory is not affected. This is the default option.
• --hard – The staged snapshot and the working directory are both updated to match the specified commit.

It’s easier to think of these modes as defining the scope of a git reset operation:

These flags are often used with HEAD as the parameter. For instance, git reset --mixed HEAD has the affect of unstaging all changes, but leaves them in the working directory. On the other hand, if you want to completely throw away all your uncommitted changes, you would use git reset --hard HEAD. These are two of the most common uses of git reset.

Be careful when passing a commit other than HEAD to git reset, since this re-writes the current branch’s history. As discussed in The Golden Rule of Rebasing, this a big problem when working on a public branch.

Checkout

By now, you should be very familiar with the commit-level version of git checkout. When passed a branch name, it lets you switch between branches.

git checkout hotfix

Internally, all the above command does is move HEAD to a different branch and update the working directory to match. Since this has the potential to overwrite local changes, Git forces you to commit or stash any changes in the working directory that will be lost during the checkout operation. Unlike git reset, git checkout doesn’t move any branches around.

You can also check out arbitrary commits by passing in the commit reference instead of a branch. This does the exact same thing as checking out a branch: it moves the HEAD reference to the specified commit. For example, the following command will check out out the grandparent of the current commit:

git checkout HEAD~2

This is useful for quickly inspecting an old version of your project. However, since there is no branch reference to the current HEAD, this puts you in a detached HEAD state. This can be dangerous if you start adding new commits because there will be no way to get back to them after you switch to another branch. For this reason, you should always create a new branch before adding commits to a detached HEAD.

Revert

Reverting undoes a commit by creating a new commit. This is a safe way to undo changes, as it has no chance of re-writing the commit history. For example, the following command will figure out the changes contained in the 2nd to last commit, create a new commit undoing those changes, and tack the new commit onto the existing project.

git checkout hotfix
git revert HEAD~2

This can be visualized as the following:

Contrast this with git reset, which does alter the existing commit history. For this reason, git revert should be used to undo changes on a public branch, and git reset should be reserved for undoing changes on a private branch.

You can also think of git revert as a tool for undoing committed changes, while git reset HEAD is for undoing uncommitted changes.

Like git checkout, git revert has the potential to overwrite files in the working directory, so it will ask you to commit or stash changes that would be lost during the revert operation.

File-level Operations

The git reset and git checkout commands also accept an optional file path as a parameter. This dramatically alters their behavior. Instead of operating on entire snapshots, this forces them to limit their operations to a single file.

Reset

When invoked with a file path, git reset updates the staged snapshot to match the version from the specified commit. For example, this command will fetch the version of foo.py in the 2nd-to-last commit and stage it for the next commit:

git reset HEAD~2 foo.py

As with the commit-level version of git reset, this is more commonly used with HEAD rather than an arbitrary commit. Running git reset HEAD foo.py will unstage foo.py. The changes it contains will still be present in the working directory.

The --soft, --mixed, and --hard flags do not have any effect on the file-level version of git reset, as the staged snapshot is always updated, and the working directory is never updated.

Checkout

Checking out a file is similar to using git reset with a file path, except it updates the working directory instead of the stage. Unlike the commit-level version of this command, this does not move the HEAD reference, which means that you won’t switch branches.

For example, the following command makes foo.py in the working directory match the one from the 2nd-to-last commit:

git checkout HEAD~2 foo.py

Just like the commit-level invocation of git checkout, this can be used to inspect old versions of a project—but the scope is limited to the specified file.

If you stage and commit the checked-out file, this has the effect of “reverting” to the old version of that file. Note that this removes all of the subsequent changes to the file, whereas the git revert command undoes only the changes introduced by the specified commit.

Like git reset, this is commonly used with HEAD as the commit reference. For instance, git checkout HEAD foo.py has the effect of discarding unstaged changes to foo.py. This is similar behavior to git reset HEAD --hard, but it operates only on the specified file.

Summary

You should now have all the tools you could ever need to undo changes in a Git repository. The git reset, git checkout, and git revert commands can be confusing, but when you think about their effects on the working directory, staged snapshot, and commit history, it should be easier to discern which command fits the development task at hand.

The table below sums up the most common use cases for all of these commands. Be sure to keep this reference handy, as you’ll undoubtedly need to use at least some them during your Git career.

Reproduced from here

这两天在看Java编程的书，看到String的时候将之前没有弄懂的都理清了一遍，本来想将String之间的区别记录下来的，在找资料的时候发现这位网友整理的很不错，值得借鉴。我就在这个上面添一点自己的理解了。原文地址在上面。

关于这三个类在字符串处理中的位置不言而喻，那么他们到底有什么优缺点，到底什么时候该用谁呢？下面我们从以下几点说明一下:

1.三者在执行速度方面的比较：StringBuilder > StringBuffer > String

2.String < (StringBuffer，StringBuilder)的原因

String：字符串常量
StringBuffer：字符串变量
StringBuilder：字符串变量

从上面的名字可以看到，String是“字符串常量”，也就是不可改变的对象。对于这句话的理解你可能会产生这样一个疑问，比如这段代码：

String s = "abcd";
s = s + 1;
System.out.print(s);// result: abcd1

我们明明就是改变了String型的变量s的，为什么说是没有改变呢? 其实这是一种欺骗，JVM是这样解析这段代码的：首先创建对象s，赋予一个abcd，然后再创建一个新的对象s用来执行第二行代码，也就是说我们之前对象s并没有变化，所以我们说String类型是不可改变的对象了，由于这种机制，每当用String操作字符串时，实际上是在不断的创建新的对象，而原来的对象就会变为垃圾被ＧＣ回收掉，可想而知这样执行效率会有多底。String类中每一个看起来会修改String值的方法，实际上都是创建一个全新的String对象，已包含修改后的字符串，而最初的String对象则丝毫未动。

而StringBuffer与StringBuilder就不一样了，他们是字符串变量，是可改变的对象，每当我们用它们对字符串做操作时，实际上是在一个对象上操作的，这样就不会像String一样创建一些而外的对象进行操作了，当然速度就快了。StringBuffer和String有很多相似之处，但是其内部的实现却有很大的差别，StringBuffer其实是一个分装一个字符数组，同时提供了对这个字符数组的相关操作。StringBuffer()构造一个字符缓冲区，其初始容量为16个字符。

3.一个特殊的例子：

String str = "This is only a" + "simple" + "test";
StringBuffer builder = new StringBuilder("This is only a").append("simple").append("test");

你会很惊讶的发现，生成str对象的速度简直太快了，而这个时候StringBuffer居然速度上根本一点都不占优势。其实这是JVM的一个把戏，实际上String str = "This is only a" + " simple" + "test";其实就是String str = "This is only a simple test";

所以不需要太多的时间了。但大家这里要注意的是，如果你的字符串是来自另外的String对象的话，速度就没那么快了，譬如：

String str2 = "This is only a";
String str3 = "simple";
String str4 = "test";
String str1 = str2 +str3 + str4;

这时候JVM会规规矩矩的按照原来的方式去做。

4.StringBuilder与StringBuffer

StringBuilder：线程非安全的
StringBuffer：线程安全的

当我们在字符串缓冲区被多个线程使用时，JVM不能保证StringBuilder的操作是安全的，虽然他的速度最快，但是可以保证StringBuffer是可以正确操作的。当然大多数情况下就是我们是在单线程下进行的操作，所以大多数情况下是建议用StringBuilder而不用StringBuffer的，就是速度的原因。

对于三者使用的总结：

1.如果要操作少量的数据用 = String

2.单线程操作字符串缓冲区下操作大量数据 = StringBuilder

3.多线程操作字符串缓冲区下操作大量数据 = StringBuffer.

关于三者的速度，自己写了个测试代码：

package com.wt.others;

public class StringCompare {
    public static void main(String[] args) {
        // TODO Auto-generated method stub
        String text = "";
        long beginTime = 0l;
        long endTime = 0l;
        StringBuffer buffer = new StringBuffer();
        StringBuilder builder = new StringBuilder();
        beginTime = System.currentTimeMillis();
        for(int i=0; i<20000; i++){
            buffer.append(String.valueOf(i));
        }
        endTime = System.currentTimeMillis();
        System.out.println("StringBuffer time is : "+ (endTime - beginTime));

        beginTime = System.currentTimeMillis();
        for(int i=0; i<20000; i++){
            builder.append(String.valueOf(i));
        }
        endTime = System.currentTimeMillis();
        System.out.println("StringBuilder time is : "+ (endTime - beginTime));

        beginTime = System.currentTimeMillis();
        for(int i=0; i<20000; i++){
            text = text + i;
        }
        endTime = System.currentTimeMillis();
        System.out.println("String time is : "+ (endTime - beginTime));
    }
}

运行结果可以直观的看出：

StringBuffer time is : 5
StringBuilder time is : 3
String time is : 1550

如果将20000改为100，结果为：

StringBuffer time is : 1
StringBuilder time is : 0
String time is : 1

还是可以直观看出在单线程使用时，StringBuilder速度很快。

以下是最近看的一本书《Google软件测试之道》里的一些摘录，收获很多。

1. 讨论测试开发比并没有什么意义，如果你是一名开发人员，同时也是一名测试人员，如果你的职位头衔上有测试的字样，你的任务就是使那些头衔上没有测试的人可以更好的去做测试。

2. 只有在软件产品变得重要的时候质量才显得重要。

3. 假如你被要求去实现一个函数count（void *）返回一个字符串中大写字母A出现的次数。如果候选人上来就直接开始写代码，这无非在传递一个强烈的信息，只有一件事情需要去做而我正在做这个事情，这个事情就是写代码。SET不会遵循这样的世界观，他们会先把问题搞清楚。

   这个函数是用来做什么的?我们为什么要构建它？这个函数的原型看起来正确吗？我们期望候选人可以关心函数的正确性以及如何验证期望的行为。一个问题值得更多的关注!候选人如果没头没脑地就跳进来编码，试图解决问题，在对待测试问题上他同样会没头没脑。如果我们提出一个问题是给模块增加测试场景，我们不希望候选人上就直接开始罗列所有可能的测试用例，直到我们强迫他停下来。其实我们只是希望他先执行最佳的测试用例。

   SET的时间是有限的。我们希望候选人能够回过头来寻找最有效的解决问题的方法，为先前的函数定义可以做一些改进。优秀的SET在面对拙劣的API定义的情况下，在测试的过程中也可以把这个API定义变得更漂亮些。

   普通的候选人会花几分钟通过提问题和陈述的方式来理解需求文档，例如以下几点。

   • 传入的字符串编码是什么：ASCII, UTF-8或其他的编码方式?
   • 函数名字比较槽糕，应该是驼峰式(CamelCased)的?需要更多说明描述，还是这里应该遵循其他的什么命名规范?
   • 返回值类型是什么(或许面试官忘记了，所以我会增加一个int类型的返回值在函数原型之前)?
   • 如果只有一个A的情况，计数结果是多少?它对小写字母a也计数吗?

   • 在标准库中不是已经有这样的函数了吗(为了面试的目的，假装你是第一个实现这个函数功能的人)?

     更好的候选人则会考虑的更多一些。

   • 考虑下扩展性：或许返回值的类型应该是1个64位的整形，因为Google经常涉及海量数据。
   • 考虑下复用性：为什么这个函数是针对大写字母A进行计数的?一个好的办法是参数化，使得任意的字符都可以被计数，而不是使用不同的函数来实现。

   • 考虑下安全性：这此指针都是来自于可信任的地址吗?

     最佳的候选人会这样考虑。

   • 考虑扩展

     这个函数会在Shared data(译注:数据分区，是数据库存储分割(partition)的一种方式。水平分割是一个数据库的设计一准则，数据以记录行的方式存储在不同的物理位置，而不是通过不同列的方式存储。或许这才是调用这个函数最有用的形式。在这个场景需要考虑一些什么问题吗?针对整个互联网的所有文档运行这个函数，该如何考虑性能和正确性?

     如果这个子程序被每一个Google查询所调用，而且由于外部的封装层面已经对参数做了验证，传递的指针是安全的，或许减少1个空指针的检查会每天节省上亿次的cpu调用周期，井缩短用户的响应时间。最少要理解全部参数验证所带来的潜在影响。

   • 考虑基于常量的优化

     我们可以假设输入的数据是已经排好顺序的吗？如果是那样，我们或许可以在找到第个大写字母B之后就快速退出。输入的数据是什么结构?多数情况下都是A吗?多数是字符的混合，还是只包含字毋A和空格？如果那样，在我们比较指令的地方或许可以做些优化。当在处理大数据，甚至小数据的时候，在代码执行的时候对于真实的计算延迟也会有比较显著的亚线性变化。

   • 考虑安全性

     在许多系统上，如果这是一段对于安全敏感的代码，可以考虑更多的非空的指针做测试，在某些系统上，1是一个非法指针。

     增加一个字符长度的参数，用以保证代码不会运行到指定字符串之外的部分。检查字符串长度，这个参数的值是否正常。那些不是以null结尾的字符串是黑客们的最爱。

     如果指针指向的数据能被其他的线程修改，这里就有潜在的线程安全问题。

     我们是否应该使用try/catch来捕获异常的发生？或者如果未能如预期那样正常的调用代码，我们或许应该返回错误代码给调用者。如果有错误代码的话，这些代码经过良好的定义并有文档吗？这意味着候选人在思考大型代码库和运行时刻的上下文环境方面的问题，这样的思索可以避免错误代码的重复和遗漏。

4. 在Google，如果测试运行失败需要清除的知道测试代码在做什么，否则这个测试就应该被禁止掉，或者被标记为怪异的测试，或是忽略这个测试的运行失败，这个问题如果发生了，这是编写出坏代码的SWE的责任，或是代码审查时给予通过的投票的SWE或SET的责任。

5. 使用白盒测试知道哪些用例是无效的：

   通常情况下，普通的候选人会这样做。

   • 他们会比较有条理地或体系化地提供特定的字符串(如不同的字符串大小)而不是随机的字符串。

   • 专注于产生有意义的测试数据。考虑如何去运行大型测试和使用真实环境的数据做测试。

     更优秀的候选人会这样做的更多一些。

   • 在并发线程中调用这个函数，去查看在串扰(cross talk)、死锁和内存泄露方面是否存在问题。
   • 构建长时间持续运行的测试场景。例如在一个while(true)循环中调用函数，并确保他们在不间断地长时间运行过程中保持功能正常。
   • 在构建测试用例、测试数据的产生方法、验证和执行上保持浓厚的兴趣。

6. Selenium在浏览器内部使用JavaScript实现，而WebDriver使用浏览器本身的API集成到浏览器内部。两种方法各有优劣。例如，Selenium可以在瞬间打开一个新的Chrome浏览器，但却不能上传文件或者很好地处理用户交互，因为它是JavaScript实现，必须限定在JS沙箱之内。由于WebDriver构建在浏览器里面，它可以突破这些限制，但打开个新的浏览器却比较痛苦。在我们都开始为Google工作的时候，我们决定把这两个集成到一起。

7. 风险分析

   • 哪些事件需要担心
   • 这些事件发生的可能性有多大?
   • 一旦发生，对公司产生多大影响?
   • 一旦发生，对客户产生多大影响?
   • 产品具备什么缓解措施?
   • 这些缓解措施有多大可能会失败?
   • 处理这些失败的成本有哪些？
   • 恢复过程有多困难?
   • 事件是一次性问题，还是会再次发生?

8. 对于一个web测试页面，一个文本输入框，一个计数按钮，用于计算一个字符串中大写字母A出现的个数。请设计出一系列字符串来测试这个web页面。

   一些候选人头扎进去开始罗列测试用例，这往往是一个危险的信号，说明他们还没有充分思考这个问题。根据我们的经验，追求数量而非质量的倾向，是一种低效的工作方式，因此会给负面评价。通过观察候选人在找到答案之前思考和解决问题的方式，能了解他们很多东西。

   更好的是那些会提出一些问题，来做进一步澄清的候选人：大写还是小写？只是英语吗？计算完成后文本会被清除吗？多次按下按钮会发生什么事情？诸如此类。在问题被澄清之后，候选人开始列举测试用例。重点观察他们是否使用一些疯狂的做法。他们只是在试图破坏软件，还是同时在验证它能正常工作？他们知道这两者的区别吗？他们是否能从最显而易见的简单的输入开始，尽快地发现大bug?他们能清晰地列出测试计划或数据吗?在白板上随机摆放字符串不能反映出思路的清晰性，他们很可能毫无测试计划，或者只有很粗糙很随意的测试计划。

   一个典型的列表如下：

   • “banana”: 3（一个合法的英文字）
   • “A”和”a”: 1（一个简单的有正常结果的合法输入）
   • “”: 0（一个简单的结果为0的合法输入）
   • Null: 0（简单的错误输入）
   • “AA”和”aa”: 2（个数>1并且所有字母都是A的输入）
   • “b”: 0（一个简单的非空合法输入，结果是0）
   • “aba”: 2（目标字符出现在开头和结尾，以寻找循环边界错误）
   • “bab”: 1（目标字符出现在中间）
   • space/tabs等: N（空白字符与N个A的混合）
   • 不包含A的长字符串: N, 其中N>0
   • 包含A的长字符串: N, 其中N是A出现的个数
   • X\nX字符串: N, 其中N是A出现的个数（格式化字符串）

   • {java/C/HTML/JavaScript}: N, 其中N是A出现的个数（可执行字符）

     无论丢失上述测试和总结哪几个都是一个不好的征兆。

     更好的候选人会超越输入选择，讨论更加高级的测试问题。他们可能会做以下的事情。

   • 质疑界面的外观、调色板和对比度。如“这些与相关应用风格一致吗?”，“视力困难的人能使用么？”等
   • 担心文本框太小了，建议加长以便显示更长的输入字符串。
   • 考虑这个应用能否在同一台服务器运行多个实例。会发生多个用户的串扰吗？
   • 提出疑问“数据会被记录吗”，输入串可能包含地址或其他身份信息。
   • 建议使用真实数据进行自动化测试，如从词典或书本里选择。
   • 提出疑问，“计算足够快吗?在大负载下呢?”
   • 提出疑问，“该页是可发现的吗？用户怎么能找到该页面呢？”
   • 输入HTML和JavaScript，看是否会破坏页面渲染。
   • 询问是对大写还是小写的A计数，还是都包括。

   • 尝试复制和粘贴字符串。

     还有一些想法更加高级，反映了富有经验的、宝贵的测试思维，能够比问题走的更远。他们可能会这样做。

   • 意识到计算会通过URL-encoded HTTP GET请求传递到服务器，字符串可能会在穿越网络时被截断。因此，无法保证支持多长的URL
   • 建议将此应用参数化。为何只对字母A一计数呢?
   • 考虑计算其他语台中的A(如埃A或变音符号)。
   • 考虑该应用是否可以被国际化。
   • 考虑编写脚本或者手工采样来探知字符串长度的上限(例如，通过2的指数递进算法)，然后确保在此区间内功能正常。
   • 考虑背后的实现和代码。也许有一个计数器遍历该字符串，另外一个跟踪已经遇到了多少个A累加器)。因此，可以在边界值附近变化A的个数和字符串的长度来进行测试。
   • 提出疑问，”HTTP POST方法和参数会被黑掉吗?也许有安全漏洞?”

   • 用脚本创建各种有趣的排列组合和字符串特性如长度、A的个数等的组合，自动生成测试输入和验证。

     了解候选人使用多长的字符串做为测试用例，这通常能暗示他们工作时的表现。如果候选人只是一般性的知道使用“长字符串”(最常见的答案)，但却无法就特定场景进行技术性的分析，这是一种糟糕的迹象。更懂技术的候选人，会询问字符串的规格说明，进而围绕极限点进行边界值测试。例如，当极限点是1000的时候，他们会尝试999. 1000和10010最好的候选人还会尝试2^32，以及许多其他有趣的值，例如2和10的次方。重点在于候选人表现出对真正重要的数字值的理解，而不只是使用随机数值——他们需要对底层的算法、语言、运行时和硬件都有所了解，因为这些正是错误最经常出现的地方。他们还应当基于可能的实现细节尝试不同的长度，并考虑到计数器、指针及抓环的边界错误。最优秀的候选人还会意识到系统可能是有状态的，测试必须将先前的输入考虑在内。因此，多次输入同一字符串，或者在长度为1000的字符串之后输入一个长度为0的，这些就属于重要的使用情形。

     在Google，鉴于快节奏的发布周期，规格说明经常变来变去，可以有不同的理解和修改。如果候选人能指出“5个字符的最大长度”这种描述是有点奇怪的，有可能会使用户感到疑惑，这正反映了他们能从用户角度思考。如果候选人不假思索地接受了这个描述并匆忙动手，那他们在实际工作中也很有可能如此，结果是自费力气验证了错误的行为。那些能反驳或者质疑规格说明的候选人，往往在工作中有优异的表现。当然，也要注意反驳或者质疑的方式。

9. 加入一个新项目的头几个星期，我主要用来倾听而不是发表意见，深入理解团队非常重要，要学习产品的架构，了解团队的最新动态。我不能接受一位医生在观察我不到五分钟的时间就给我开乓抗生素类的药诗之:。同样地，我也不期望个测试团队可以接受，我一开始就提出的什么解决方案。在进行诊断之前你必须先要学习。

   我感觉人们有时候做事只是因为看到别人这么做，或者他们测试某个特性的时候只是做那些他们知道怎么做的东西。如果你不问他们为什么，他们自己也不会费心思考这事儿，因为他们已经把那些作为了一种习惯。

10. 我们做的每件事都有明确的目的。我们质疑所有的事情：测试用例、每项自动化测试。其实我们正在做的很多事情就通不过这种审视。如果自动化不能带来明确的价值，我们就废弃它。所有的事情都是价值驱动的，这才能成就团队。如果要我给新晋测试经理什么建议，我会告诉他们：你们做的每一件事都要创造价值，能够持续地创造价值。

11. 我的测试人员个个都是通才。具体来说，每个人都能做手工测试，真的是每个人都能。探索式测试足深入学习理解一个产品的最佳途径。我永远不会让 一个测试开发工程师成为一个框架开发者。我希望他们深入产品并了解如何使用它。每个测试人员都必须强调用户。他们必须是专家级的用户，通晓整个产品的每个细节。在我的团队，我们把如稳定性测试、电源管理、性能测试、压力测试和第三方应用程序的快速检查都留给自动化测试完成。举个例子，没有人能够手动发现相机的内存泄露或在各个平台下验证一个单一特性的功能——这丝都需要自动化。大量重复性的工作不适合手工测试，或者一些需要机器才能达到的高精度测试就必须通过自动化测试来完成。

12. Google的测试流程可以非常简练地概括为：让每个工程师都注重质量。只要大家诚实认真地这么做，质量就会提高。代码质量从一开始就能更好，早期构建版本的质量会更高，集成也不再是必须的，系统测试可以关注于真正面向用户的问题。所有的工程师和项目都能从堆积如山的bug中解脱出来。

13. Google在测试方面的秘方：（James）那就是测试人员所拥有的技术能力(包括计算机科学的专业文凭)、测试资源的稀缺从而获得开发人员帮助和不断进行测试优化、优先考虑自动化(这样才能让人去做那些计算机做不好的事情)，以及快速迭代、集成和获得用户反馈的能力。其他公司要想效仿Google的做法，应该从这四个方面做起：技能、稀缺性、自动化和迭代集成。这就是Google测试的“秘方”，照方抓药吧！

14. 任何角色都不应被过分强调。团队的每个人都是在为产品工作，而不是为了开发过程中的某个部分。开发过程本身就是为产品服务的。除了做出更好的产品，流程的存在还有其他的目的吗？用户爱上的是产品，而不是开发产品流程。

15. 测试的价值是在于测试的动作，而不是测试产物。

    相对于被测代码来说，测试工程师生成的测试产物都是次要的：测试用例是次要的：测试计划是次要的：bug报告是次要的。这些产物都需要通过测试活动才能体现价值。不幸的是，我们过分称赞这此产物(比如在年度评估时，统计测试上程师提交的bug数量) ,而忘记了被测的软件。所有测试产物的价值，在于它们对代码的影响，进而通过产品来体现。

    独立的测试团队，倾向于把重点放在建设和维护测试产物上。如果把测试的目标定位在产品的源码上，整个产品都将受益。因此，测试人员必须把产品放在第一位。

What is Exception Wrapping?

Exception wrapping is wrapping is when you catch an exception, wrap it in a different exception and throw that exception. Here is an example:

try{
    dao.readPerson();
} catch (SQLException sqlException) {
    throw new MyException("error text", sqlException);
}

The method dao.readPerson() can throw an SQLException. If it does, the SQLException is caught and wrapped in a MyException. Notice how the SQLException (the sqlException variable) is passed to the MyException’s constructor as the last parameter.

Exception wrapping is a standard feature in Java since JDK 1.4. Most (if not all) of Java’s built-in exceptions has constructors that can take a “cause” parameter. They also have a getCause() method that will return the wrapped exception.

Why Use Exception Wrapping?

The main reason one would use exception wrapping is to prevent the code further up the call stack from having to know about every possible exception in the system. There are two main reasons for this.

The first reason is, that declared exceptions aggregate towards the top of the call stack. If you do not wrap exceptions, but instead pass them on by declaring your methods to throw them, you may end up with top level methods that declare many different exceptions. Declaring all these exceptions in each method back up the call stack becomes tedious.

The second reason is that you may not want your top level components to know anything about the bottom level components, nor the exceptions they throw. For instance, the purpose of DAO interfaces and implementations is to abstract the details of data access away from the rest of the application. Now, if your DAO methods throw SQLException’s then the code using the DAO’s will have to catch them. What if you change to an implementation that reads the data from a web service instead of from a database? Then you DAO methods will have to throw both RemoteException and SQLException. And, if you have a DAO that reads data from a file, you will need to throw IOException too. That is three different exceptions, each bound to their own DAO implementation.

To avoid this your DAO interface methods can throw DaoException. In each implementation of the DAO interface (database, file, web service) you will catch the specific exceptions (SQLException, IOException, RemoteException), wrap it in a DaoException, and throw the DaoException. Then code using the DAO interface will only have to deal with DaoException’s. It does not need to know anything about what data access technology is used in the various implementations.

Introduction

I finally decided to write up a little something about Java’s parameter passing. I’m really tired of hearing folks (incorrectly) state “primitives are passed by value, objects are passed by reference”.

I’m a compiler guy at heart. The terms “pass-by-value” semantics and “pass-by-reference” semantics have very precise definitions, and they’re often horribly abused when folks talk about Java. I want to correct that… The following is how I’d describe these

Pass-by-value

The actual parameter (or argument expression) is fully evaluated and the resulting value is copied into a location being used to hold the formal parameter’s value during method/function execution. That location is typically a chunk of memory on the runtime stack for the application (which is how Java handles it), but other languages could choose parameter storage differently.

Pass-by-reference

The formal parameter merely acts as an alias for the actual parameter. Anytime the method/function uses the formal parameter (for reading or writing), it is actually using the actual parameter.

Java is strictly pass-by-value, exactly as in C. Read the Java Language Specification (JLS). It’s spelled out, and it’s correct. In here:

When the method or constructor is invoked (�15.12), the values of the actual argument expressions initialize newly created parameter variables, each of the declared Type, before execution of the body of the method or constructor. The Identifier that appears in the DeclaratorId may be used as a simple name in the body of the method or constructor to refer to the formal parameter.

[In the above, values is my emphasis, not theirs]

In short: Java has pointers and is strictly pass-by-value. There’s no funky rules. It’s simple, clean, and clear. (Well, as clear as the evil C++-like syntax will allow ;)

Note: See the note at the end of this article for the semantics of remote method invocation (RMI). What is typically called “pass by reference” for remote objects is actually incredibly bad semantics.

The Litmus Test

There’s a simple “litmus test” for whether a language supports pass-by-reference semantics:

Can you write a traditional swap(a,b) method/function in the language?

A traditional swap method or function takes two arguments and swaps them such that variables passed into the function are changed outside the function. Its basic structure looks like

Figure 1: (Non-Java) Basic swap function structure

swap(Type arg1, Type arg2) {
    Type temp = arg1;
    arg1 = arg2;
    arg2 = temp;
}

If you can write such a method/function in your language such that calling

Figure 2: (Non-Java) Calling the swap function

Type var1 = ...;
Type var2 = ...;
swap(var1,var2);

actually switches the values of the variables var1 and var2, the language supports pass-by-reference semantics.

For example, in Pascal, you can write

Figure 3: (Pascal) Swap function

procedure swap(var arg1, arg2: SomeType);

    var
        temp : SomeType;
    begin
        temp := arg1;
        arg1 := arg2;
        arg2 := temp;
    end;

...

{ in some other procedure/function/program }

var
    var1, var2 : SomeType;

begin
    var1 := ...; { value "A" }
    var2 := ...; { value "B" } 
    swap(var1, var2);
    { now var1 has value "B" and var2 has value "A" }
end;

or in C++ you could write

Figure 4: (C++) Swap function

void swap(SomeType& arg1, Sometype& arg2) {
    SomeType temp = arg1;
    arg1 = arg2;
    arg2 = temp;
}

...

SomeType var1 = ...; // value "A"
SomeType var2 = ...; // value "B"
swap(var1, var2); // swaps their values!
// now var1 has value "B" and var2 has value "A"

(Please let me know if my Pascal or C++ has lapsed and I’ve messed up the syntax…)

But you cannot do this in Java!

Now the details…

The problem we’re facing here is statements like

In Java, Objects are passed by reference, and primitives are passed by value.

This is half incorrect. Everyone can easily agree that primitives are passed by value; there’s no such thing in Java as a pointer/reference to a primitive.

However, Objects are not passed by reference. A correct statement would be Object references are passed by value.

This may seem like splitting hairs, bit it is far from it. There is a world of difference in meaning. The following examples should help make the distinction.

In Java, take the case of

Figure 5: (Java) Pass-by-value example

public void foo(Dog d) {
    d = new Dog("Fifi"); // creating the "Fifi" dog
}

Dog aDog = new Dog("Max"); // creating the "Max" dog
// at this point, aDog points to the "Max" dog
foo(aDog);
// aDog still points to the "Max" dog

the variable passed in (aDog) is not modified! After calling foo, aDog still points to the “Max” Dog!

Many people mistakenly think/state that something like

Figure 6: (Java) Still pass-by-value…

public void foo(Dog d) { 
    d.setName("Fifi");
}

shows that Java does in fact pass objects by reference.

The mistake they make is in the definition of

Figure 7: (Java) Defining a Dog pointer

Dog d;

itself. When you write that definition, you are defining a pointer to a Dog object, not a Dog object itself.

On Pointers versus References…

The problem here is that the folks at Sun made a naming mistake.

In programming language design, a “pointer” is a variable that indirectly tracks the location of some piece of data. The value of a pointer is often the memory address of the data you’re interested in. Some languages allow you to manipulate that address; others do not.

A “reference” is an alias to another variable. Any manipulation done to the reference variable directly changes the original variable.

Check out the second sentence of http://java.sun.com/docs/books/jls/third_edition/html/typesValues.html#4.3.1.

“The reference values (often just references) are pointers to these objects, and a special null reference, which refers to no object”

They emphasize “pointers” in their description… Interesting…

When they originally were creating Java, they had “pointer” in mind (you can see some remnants of this in things like
NullPointerException).

Sun wanted to push Java as a secure language, and one of Java’s advantages was that it does not allow pointer arithmetic as C++ does.

They went so far as to try a different name for the concept, formally calling them “references”. A big mistake and it’s caused even more confusion in the process.

There’s a good explanation of reference variables at http://www.cprogramming.com/tutorial/references.html. (C++ specific, but it says the right thing about the concept of a reference variable.)

The word “reference” in programming language design originally comes from how you pass data to subroutines/functions/procedures/methods. A reference parameter is an alias to a variable passed as a parameter.

In the end, Sun made a naming mistake that’s caused confusion. Java has pointers, and if you accept that, it makes the way Java behaves make much more sense.

Calling Methods

Calling

Figure 8: (Java) Passing a pointer by value

foo(d);

passes the value of d to foo; it does not pass the object that d points to!

The value of the pointer being passed is similar to a memory address. Under the covers it may be a tad different, but you can think of it in exactly the same way. The value uniquely identifies some object on the heap.

However, it makes no difference how pointers are implemented under the covers. You program with them exactly the same way in Java as you would in C or C++. The syntax is just slightly different (another poor choice in Java’s design; they should have used the same -> syntax for de-referencing as C++).

In Java,

Figure 9: (Java) A pointer

Dog d;

is exactly like C++’s

Figure 10: (C++) A pointer

Dog *d;

And using

Figure 11: (Java) Following a pointer and calling a method

d.setName("Fifi");

is exactly like C++’s

Figure 12: (C++) Following a pointer and calling a method

d->setName("Fifi");

To sum up: Java has pointers, and the value of the pointer is passed in. There’s no way to actually pass an object itself as a parameter. You can only pass a pointer to an object.

Keep in mind, when you call

Figure 13: (Java) Even more still passing a pointer by value

foo(d);

you’re not passing an object; you’re passing a pointer to the object.

For a slightly different (but still correct) take on this issue, please see this. It’s from Peter Haggar’s excellent book, Practical Java.)

A Note on Remote Method Invocation (RMI)

When passing parameters to remote methods, things get a bit more complex. First, we’re (usually) dealing with passing data between two independent virtual machines, which might be on separate physical machines as well. Passing the value of a pointer wouldn’t do any good, as the target virtual machine doesn’t have access to the caller’s heap.

You’ll often hear “pass by value” and “pass by reference” used with respect to RMI. These terms have more of a “logical” meaning, and really aren’t correct for the intended use.

Here’s what is usually meant by these phrases with regard to RMI. Note that this is not proper usage of “pass by value” and “pass by reference” semantics:

RMI Pass-by-value

The actual parameter is serialized and passed using a network protocol to the target remote object. Serialization essentially “squeezes” the data out of an object/primitive. On the receiving end, that data is used to build a “clone” of the original object or primitive. Note that this process can be rather expensive if the actual parameters point to large objects (or large graphs of objects).

This isn’t quite the right use of “pass-by-value”; I think it should really be called something like “pass-by-memento”. (See “Design Patterns” by Gamma et al for a description of the Memento pattern).

RMI Pass-by-reference

The actual parameter, which is itself a remote object, is represented by a proxy. The proxy keeps track of where the actual parameter lives, and anytime the target method uses the formal parameter, another remote method invocation occurs to “call back” to the actual parameter. This can be useful if the actual parameter points to a large object (or graph of objects) and there are few call backs.

This isn’t quite the right use of “pass-by-reference” (again, you cannot change the actual parameter itself). I think it should be called something like “pass-by-proxy”. (Again, see “Design Patterns” for descriptions of the Proxy pattern).

Follow up from stackoverflow.com

I posted the following as some clarification when a discussion on this article arose on http://stackoverflow.com.

The Java Spec says that everything in java is pass-by-value. There is no such thing as “pass-by-reference” in java.

The key to understanding this is that something like

Figure 14: (Java) Not a Dog; a pointer to a Dog

Dog myDog;

is not a Dog; it’s actually a pointer to a Dog.

What that means, is when you have

Figure 15: (Java) Passing the Dog’s location

Dog myDog = new Dog("Rover");
foo(myDog);

you’re essentially passing the address of the created Dog object to the foo method. (I say essentially b/c java pointers aren’t direct addresses, but it’s easiest to think of them that way)

Suppose the Dog object resides at memory address 42. This means we pass 42 to the method.

If the Method were defined as

Figure 16: (Java) Looking at the called method in detail

public void foo(Dog someDog) {
    someDog.setName("Max");     // AAA
    someDog = new Dog("Fifi");  // BBB
    someDog.setName("Rowlf");   // CCC
}

Let’s look at what’s happening.

the parameter someDog is set to the value 42

at line “AAA”

someDog is followed to the Dog it points to (the Dog object at address 42) that Dog (the one at address 42) is asked to change his name to Max

at line “BBB”

a new Dog is created. Let’s say he’s at address 74 we assign the parameter someDog to 74

at line “CCC”

someDog is followed to the Dog it points to (the Dog object at address 74) that Dog (the one at address 74) is asked to change his name to Rowlf then, we return

Now let’s think about what happens outside the method:

Did myDog change?

There’s the key.

Keeping in mind that myDog is a pointer, and not an actual Dog, the answer is NO. myDog still has the value 42; it’s still pointing to the original Dog.

It’s perfectly valid to follow an address and change what’s at the end of it; that does not change the variable, however.

Java works exactly like C. You can assign a pointer, pass the pointer to a method, follow the pointer in the method and change the data that was pointed to. However, you cannot change where that pointer points.

In C++, Ada, Pascal and other languages that support pass-by-reference, you can actually change the variable that was passed.

If Java had pass-by-reference semantics, the foo method we defined above would have changed where myDog was pointing when it assigned someDog on line BBB.

Think of reference parameters as being aliases for the variable passed in. When that alias is assigned, so is the variable that was passed in.

TestNG Listeners are fundamental in setting up and managing our integration with Selenium driver behind the scenes of a test. The issue we may run into from time to time is the driver initialization may fail and unless you suppress the exception this will cause a configuration failure in testNG.

Below I will attempt to explain some things you can do to overcome the configuration failure and help you better handle it in your framework.

alwaysRun

When configuration failures occur testNG’s default behavior is to then skip every after listener down the line. For instance if your exception occurred in beforeClass or beforeMethod then afterMethod and afterClass will be skipped. You can solve the issue of subsequent after listeners not being executed by adding the attribute alwaysRun=true to the annotation of the listeners. This could be useful if you are collecting reporting information during your execution and your rely on these listeners to set some data for your reporting.

Remember to create flags to verify if any additional cleanup should be performed in the event that the before listeners DID fail. For instance if the driver was never created calling driver quit could throw another exception in your afterMethod execution.

@AfterMethod(alwaysRun = true)
    public void afterMethod(Method method) {

Should you add alwaysRun to your before listeners?

If you are adding group names to your methods you probably should but that is entirely up to you. When adding alwaysRun to before listeners it tells the listener that it is ok to run even if it doesn’t contain the group name for the test specified in your testng.xml suite.

@BeforeMethod(alwaysRun = true)
    public void beforeMethod(Method method)

If you wish to have a different set of before listeners which perform different behaviors based on the group name then you shouldn’t use alwaysRun.

configFailurePolicy

The default behavior of the configFailurePolicy setting has changed, purposely or not, from the behavior of “continue” to the behavior of “skip” over time. What this controls is whether the test methods should attempt to be ran regardless of a configuration failure. When set to skip the methods are skipped, when set to continue the methods attempt to run anyways.

In general you likely want your tests to skip because there is no need to attempt to run them if the driver has failed to initialize, to do so would only lead to additional failures and waste additional execution time. To force the setting of skip add the configfailurepolicy="skip" to the the suite tag of your testng.xml.

However if you wish to have the execution continue on for some reason you can change this to configfailurepolicy="continue".

As of this writing the most current version of testNG is 6.9.10 and its default configfailurepolicy (what happens when this attribute is omitted from your testng.xml) is set to skip.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite" parallel="tests" thread-count="10" verbose="10" configfailurepolicy="skip">
  <listeners>
    <listener class-name="org.uncommons.reportng.HTMLReporter"/>
    <listener class-name="org.uncommons.reportng.JUnitXMLReporter"/>
    <listener class-name="utilities.TestListener" />
  </listeners>
  <test name="Test Chrome implicitNotVisible">
  <parameter name="targetEnvironment" value="Chrome" />
    <parameter name="network" value="" />
    <parameter name="networkLatency" value="" />
    <classes>
      <class name="AmazonTesting.SleepTestSystem">
        <methods>
          <include name="implicitNotVisible" />
        </methods>
      </class>
    </classes>
  </test>
</suite>

This is really useful when I want to:

• Compare code from two branches in IDE instead of a GUI tool like smartgit.
• Perform a long running test on one branch and working on another branch.

See this github blog post and official documentation for reference.

For me there isn’t really a huge difference between the two and in some establishments the terms could be used interchangeably. Everywhere is different. I would try and explain it like so:

System testing: You’re testing the whole system i.e. all of it’s components to ensure that each is functioning as intended. This is more from a functional side to check against requirements.

End to end testing: This is more about the actual flow through a system in a more realistic end user scenario. Can a user navigate the application as expected and does it work. You’re testing the workflow.

For example if you were to test an e-commerce site the shop front, browsing for items, cart and checkout would all work fine in systems test. You may then find issues with the workflow of moving between these areas of functionality in an end to end test.

Reference:

• System Testing
• End To End Testing

By Chuan Ji

Nov 21, 2012

First things first — this is not about how to apply a rooting method, e.g., a one-click-root, to an Android device. Rather, it is about how one could go about developing a rooting method for a device that no one has rooted before, and is told through my experiences with rooting a particular device — the Barnes & Noble Nook Tablet 8GB. For context, you can read my original thread “Root for Nook Tablet 8GB (w/ Android Market) on XDA-Developers where I published my rooting method, which has reached a download count of — wait for it — OVER NINE THOUSAND!!!

For an overview of how rooting works behind the scenes, you may want to read my previous article How Rooting Works - A Technical Explanation of the Android Rooting Process as background.

Sometime in late February (2012), on a visit to to a Barnes & Noble store in Boston, I bought the then freshly released Nook Tablet 8GB entirely on impulse for $199. Being the hax0r that I am, the first thing I did when I got home was to try to root the device. It came as a nasty surprise, therefore, when I discovered that no one had yet succeeded in rooting the device. All I could find was a YouTube video showing that the existing rooting method for its cousin, the Nook Tablet 16GB, did not work. After waiting for a few days, the absolutely pathetic app store and handicaps instituted by B&N finally motivated me to develop a rooting method for the device myself.

The Plan

So, how do you go about rooting an Android device? As I explain in my previous article, rooting is basically a two-step process:

1. Find exploit that allows execution of arbitrary code as root
2. Use exploit to install su (with SUID bit set) and Superuser.apk as root

After su and Superuser.apk are installed correctly, apps that require root (such as Titanium Backup or AdAway) will invoke su to run code as the privileged user.

The Process

There are many generic or device-specific exploits that a hax0r may leverage to achieve privileged execution of arbitrary code. I would again refer you to this excellent presentation on various Android root exploits that have been or may still be used for this purpose.

However, none of these methods that I knew of could work on the Nook Tablet, which is probably one of the most locked-down Android ROMs out there:

• Bootloader is locked.
• ADB is disabled, and cannot be enabled from the UI.
• Installing non-market apps (raw APKs) is disabled, and cannot be enabled from the UI.
• No access to Google Play / Android Market (or any Google Apps).

This rules out

1) root APKs and
2) the majority of exploits out there that require executing commands over ADB. It means that one cannot run any code on the device that does not come from B&N period.

But of course there was another way in. Somebody on XDA-Developers had discovered that the bootloader of the Nook Tablet supported booting off an Android system located in partition images stored on an external microSD card. This mechanism is probably used to repair corrupted system partitions by B&N customer support.

The solution, then, is clear: we create dummy system partition images that, instead of booting an Android system, installs su and Superuser.apk into the “normal” Android system in internal flash memory. More concretely, I modified the system initialization file inside the initrd inside the boot partition image to invoke a custom script that copied the relevant files into the system partition in the internal flash memory.

I based my work on bauwks’s 2nduboot images and used abootimg to unpack files in the boot partition image boot.img:

# Extracts files in the boot partition image into the current directory.
abootimg -x ./boot.img
# Extract files in the initrd cpio archive into the folder ./ramdisk/
aboot-unpack-initrd ./initrd.img

I changed the system initialization file init.omap4430.rc in the initrd to mount the system partition of the internal flash memory at /foo rather than /system, because later system initialization steps attempt to remount /system as read-only and so on, and for some reason I could not disable that behavior:

on fs
    mkdir /foo
    mount ext4 /dev/block/platform/mmci-omap-hs.1/by-name/system /foo wait

I added the following to the system initialization file init.rc in the initrd to start my rooting script:

service root_script /sbin/busybox ash /assets/run.sh
    oneshot

My rooting script, which I place in the directory assets in the initrd, installs not only su, but also Google Play and other Google apps which are missing from the Nook Tablet ROM:

# Install su and Superuser.apk
/sbin/busybox cp /assets/su /foo/bin/
/sbin/busybox cp /assets/su /foo/xbin/
/sbin/busybox chmod 06755 /foo/xbin/su
/sbin/busybox chmod 06755 /foo/bin/su
/sbin/busybox cp /assets/Superuser.apk /foo/app/

# Install Busybox.
/sbin/busybox cp /sbin/busybox /foo/xbin/
/sbin/busybox chmod 06755 /foo/xbin/busybox

# Install Google Play and other Google apps.
/sbin/busybox cp /assets/*.apk /foo/app/
/sbin/busybox cp /assets/com.google.android.maps..xml /foo/etc/permissions/
/sbin/busybox cp /assets/com.google.android.maps.jar /foo/framework/
/sbin/busybox cp /assets/libvoicesearch.so /foo/lib/

# Done.
/sbin/busybox mount -o ro,remount /foo

In accordance with the above script, I had placed all assets in the assets directory inside the initrd, and the Busybox binary in sbin inside the initrd. With all changes done, I pack everything back into a new boot.img:

# Build initrd cpio archive
abootimg-pack-initrd initrd.img.new
# Build new boot.img using previously extracted components
abootimg --create boot.img.new -f ./bootimg.cfg -k ./zImage -r ./initrd.img.new

Then I replace the boot.img on the SD card image with my boot.img.new, and the rooting method is done.

Final words

The actual process, of course, was much, much more painful. The Nook Tablet’s bootloader is very picky; some microSD cards just won’t work, there is a file size limit on boot.img, etc.. It was also after much frustration that I discovered the mount-to-/foo trick. And I could not even keep track of how many factory restores I had to perform on the device to undo bad modifications. But it was still a lot of fun, and the euphoria at the end and the feeling of accomplishment when reply posts started rolling in and random people were donating $5 as a token of their appreciation could not be overstated.

Good luck rooting!

By Chuan Ji

Oct 19, 2011

I have always been curious how rooting actually worked behind the scenes. After recently acquiring a new Eee Pad Slider, a Honeycomb tablet that so far no one has been able to root, the frustration of being locked out of this amazing piece of hardware with so much potential led me to finally sit down and figure out what exactly rooting means, what it entails from a technical perspective, and how hackers out in the wild are approaching the rooting of a new device. Although all this information is out there, I have not been able to find a good article that had both the level of technical detail that I wanted and an appropriate introduction to the big picture, and so I decided to write my own.

This is NOT a noob-friendly guide to rooting a particular Android device. Rather, it is a general explanation of how stock Android ROMs try to prevent unprivileged access, how hackers attack this problem and how rooting software leverage various exploits to defeat these security mechanisms.

I. The Goal

Let us first take a step back and consider exactly what we mean by rooting. Forget flashing custom ROMs, enabling WiFi tethering or installing Superuser.apk; fundamentally, rooting is about obtaining root access to the underlying Linux system beneath Android and thus gaining absolute control over the software that is running on the device. Things that require root access on a typical Linux system — mounting and unmounting file systems, starting your favorite SSH or HTTP or DHCP or DNS or proxy servers, killing system processes, chroot-ing, etc., — require root access on Android as well. Being able to run arbitrary commands as the root user allows you to do absolutely anything on a Linux / Android system, and this is real goal of rooting.

Stock OEM Android builds typically do not allow users to execute arbitrary code as root. This essentially means that you as a user are granted only limited control over your own device; you can make your device do task X only if the manufacturer explicitly decided to allow it and shipped a program to do it. You will not be able to use third-party apps to accomplish a task that your manufacturer does not wish you to do. WiFi tethering is a good example of this. Cell phone carriers obviously do not want you to tether your phone without paying them additional charges. Therefore, many phones come pre-packaged with their own proprietary WiFi tethering apps that demand extraneous fees. But without root access, you will not be able to install a free alternative like Wireless Tether For Root Users. Why this is accepted practice in the industry is a mystery to me. The only difference between cell phones, tablets and computers is their form factor; but while a PC vendor would fail spectacularly if they tried to prevent users from running arbitrary programs on their machines, cell phone vendors are clearly not judged along the same lines. But such arguments would belong to another article.

II. The Enemy: Protection Mechanisms On A Stock OEM Android ROM

1. Bootloader and Recovery

The bootloader, the first piece of code executed when your device is powered on, is responsible for loading the Android OS and the recovery system and flashing a new ROM. People refer to some bootloaders as “unlocked” if a user can flash and boot arbitrary ROMs without hacking; unfortunately, many Android devices have locked bootloaders that you would have to hack around in order to make them do anything other than boot the stock ROM. A Samsung smartphone I had used some months ago had an unlocked bootloader; I could press a certain combination of hardware keys on the phone, connect it to my computer, and flash any custom ROM onto it using Samsung’s utilities without having to circumvent any protection mechanisms. The same is not true for my Motorola Droid 2 Global; the bootloader, as far as I know, cannot be hacked. The Eee Pad Slider, on the other hand, is an interesting beast; as with other nVidia Tegra 2 based devices, its bootloader is controllable through the nvflash utility, but only if you know the secure boot key (SBK) of the device. (The SBK is a private AES key used to encrypt the commands sent to the bootloader; the bootloader will only accept the command if it has been encrypted by the particular key of the device.) Currently, as the SBK of the Eee Pad Slider is not publicly known, the bootloader remains inaccessible.

System recovery is the second piece of low-level code on board any Android device. It is separate from the Android userland and is typically located on its own partition; it is usually booted by the bootloader when you press a certain combination of hardware keys. It is important to understand that it is a totally independent program; Linux and the Android userland is not loaded when you boot into recovery, and any high-level concept such as root does not exist here. It is simple program that really is a very primitive OS, and it has absolute control over the system and will do anything you want as long as the code to do it is built in. Stock recovery varies with the manufacturer, but often includes functionalities like reformatting the /data partition (factory reset) and flashing an update ROM (update.zip, located at the root of the external microSD card) signed by the manufacturer. Note I said signed by the manufacturer; typically it is not possible to flash custom update files unless you obtain the private key of the manufacturer and sign your custom update with it, which is both impossible for most and illegal under certain jurisdictions. However, since recovery is stored in a partition just like /system, /data and /cache (more about that later), you can replace it with a custom recovery if you have root access in Linux / Android. Most people do just that upon rooting their device; ClockworkMod Recovery is a popular third-party recovery image, and allows you to flash arbitrary ROMs, backup and restore partitions, and lots of other magic.

2. ADB

ADB (see the official documentation for ADB) allows a PC or a Mac to connect to an Android device and perform certain operations. One such operation is to launch a simple shell on the device, using the command adb shell. The real question is what user do the commands executed by that shell process run as. It turns out that it depends on the value of an Android system property, named ro.secure. (You can view the value of this property by typing getprop ro.secure either through an ADB shell or on a terminal emulator on the device.) If ro.secure=0, an ADB shell will run commands as the root user on the device. But if ro.secure=1, an ADB shell will run commands as an unprivileged user on the device. Guess what ro.secure is set to on almost every stock OEM Android build. But can we change the value of ro.secure on a system? The answer is no, as implied by the ro in the name of the property. The value of this property is set at boot time from the default.prop file in the root directory. The contents of the root directory are essentially copied from a partition in the internal storage on boot, but you cannot write to the partition if you are not already root. In other words, this property denies root access via ADB, and the only way you could change it is by gaining root access in the first place. Thus, it is secure.

3. Android UI

On an Android system, all Android applications that you can see or interact with directly are running as _un_privileged users in sandboxes. Logically, a program running as an unprivileged user cannot start another program that is run as the privileged user; otherwise any program can simply start another copy of itself in privileged mode and gain privileged access to everything. On the other hand, a program running as root can start another program as root or as an unprivileged user. On Linux, privilege escalation is usually accomplished via the su and sudo programs; they are often the only programs in the system that are able to execute the system call setuid(0) that changes the current program from running as an unprivileged user to running as root. Apps that label themselves as requiring root are in reality just executing other programs (often just native binaries packaged with the app) through su. Unsurprisingly, stock OEM ROMs never come with these su. You cannot just download it or copy it over either; it needs to have its SUID bit set, which indicates to the system that the programs this allowed to escalate its runtime privileges to root. But of course, if you are not root, you cannot set the SUID bit on a program. To summarize, what this means is that any program that you can interact with on Android (and hence running in unprivileged mode) is unable to either
1) gain privileged access and execute in privileged mode, or
2) start another program that executes in privileged mode. If this holds, the Android system by itself is pretty much immune to privilege escalation attempts.

We will see the loophole exploited by on-device rooting applications in the next section.

III. Fighting the System

So how the hell do you root an Android? Well, from the security mechanisms described above, we can figure out how to attack each component in turn.

If your device happens to have an unlocked bootloader, you’re pretty much done. An example is the Samsung phone that I had had. Since the bootloader allowed the flashing of arbitrary ROMs, somebody essentially pulled the stock ROM from the phone (using dd), added su, and repackaged it into a modified ROM. All I as a user needed to do was to power off the phone, press a certain combination of hardware keys to start the phone in flashing mode, and use Samsung’s utilities to flash the modified ROM onto the phone.

Believe it or not, certain manufacturers don’t actually set ro.secure to 1. If that is the case, rooting is even easier; just plug the phone into your computer and run ADB, and you now have a shell that can execute any program as root. You can then mount /system as read-write, install su and all your dreams have come true.

But many other Android devices have locked bootloaders and ro.secure set. As explained above, they should not be root-able because you can only interact with unprivileged programs on the system and they cannot help you execute any privileged code. So what’s the solution?

We know that a number of important programs, including low-level system services, must run as root even on Android in order to access hardware resources. Typing ps on an Android shell (either via ADB or a terminal emulator on the device) will give you an idea. These programs are started by the init process, the first process started by the kernel (I often feel that the kernel and the init process are kind of analogous to Adam and Eve — the kernel spawns init in a particular fashion, and init then goes on and spawns all other processes) which has to run as root because it needs to start other privileged system processes.

Now here’s the key insight: if you can hack / trick one of these system processes running in privileged mode to execute your arbitrary code, you have just gained privileged access to the system. This how all one-click-root methods work, including z4root, gingerbreak, and so on. If you are truly curious, I highly recommend this excellent presentation on the various exploits used by current rooting tools, but the details are not as relevant here as the simple idea behind them. That idea is that there are vulnerabilities in the system processes running as root in the background that, if exploited, will allow us to execute arbitrary code as root. Well, that “arbitrary code” is most certainly a piece of code that mounts /system in read-write mode and installs a copy of su permanently on the system, so that from then on we don’t need to jump through the hoops to run the programs we really wanted to run in the first place.

Since Android is open source as is Linux, what people have done is to scrutinize and reason about the source code of the various system services until they find a security hole they can leverage. This becomes increasingly hard as Google and the maintainers of the various pieces of code fix those particular vulnerabilities when they are discovered and published, which means that the exploits will eventually become obsolete with newer devices. But the good news is that manufacturers are not stupid enough to push OTA updates to fix a vulnerability just to prevent rooting as it is very expensive for them; in addition, devices in the market are always lagging behind the newest software releases. Thus, it takes quite some time before these rooting tools are rendered useless by new patches, and by then hopefully other exploits would have been discovered.

IV. See It In Action!

To see all of this in action, you are invited to check out my follow-up article: Android Rooting: A Developer’s Guide, which explains how I applied this stuff to figure out how to root an actual device.

By Nicola Paolucci

Developer

On May 16, 2013

Update: I wrote a follow up article on the power of Git subtree

The Internet is full of articles on why you should not use Git submodules. I mostly agree, although I am not so harsh in my evaluation. As I explained in a previous post, submodules are useful for a few use cases but have several drawbacks.

Are there alternatives? The answer is: yes! There are (at least) two tools that can help track the history of software dependencies in your project while allowing you to keep using git:

• git subtree
• google repo

In this post I will be looking at git subtree and show why it is an improvement – albeit not perfect – over git submodule.

As a working example I run to my usual use case. How do I easily store and keep up to date the vim plugins used in my dotfiles?

Why use subtree instead of submodule?

There are several reasons why you might find subtree better to use:

• Management of a simple workflow is easy.
• Older version of git are supported (even before v1.5.2).
• The sub-project’s code is available right after the clone of the super project is done.
• subtree does not require users of your repository to learn anything new, they can ignore the fact that you are using subtree to manage dependencies.
• subtree does not add new metadata files like submodules doe (i.e. .gitmodule).
• Contents of the module can be modified without having a separate repository copy of the dependency somewhere else.

In my opinion the drawbacks are acceptable:

• You must learn about a new merge strategy (i.e. subtree).
• Contributing code back upstream for the sub-projects is slightly more complicated.
• The responsibility of not mixing super and sub-project code in commits lies with you.

How to use git subtree?

git subtree is available in stock version of git available since May 2012 – 1.7.11+. The version installed by homebrew on OSX already has subtree properly wired but on some platforms you might need to follow the installation instructions.

Let me show you the canonical example of tracking a vim plug-in using git subtree.

The quick and dirty way without remote tracking

If you just want a couple of one liners to cut and paste just read this paragraph.

First add the subtree at a specified prefix folder:

git subtree add --prefix .vim/bundle/tpope-vim-surround https://bitbucket.org/vim-plugins-mirror/vim-surround.git master --squash

(The common practice is to not store the entire history of the sub-project in your main repository, but If you want to preserve it just omit the -squash flag.)

The above command produces this output:

git fetch https://bitbucket.org/vim-plugins-mirror/vim-surround.git master
warning: no common commits
remote: Counting objects: 338, done.
remote: Compressing objects: 100% (145/145), done.
remote: Total 338 (delta 101), reused 323 (delta 89)
Receiving objects: 100% (338/338), 71.46 KiB, done.
Resolving deltas: 100% (101/101), done.
From https://bitbucket.org/vim-plugins-mirror/vim-surround.git
* branch            master     -} FETCH_HEAD
Added dir '.vim/bundle/tpope-vim-surround'

As you can see this records a merge commit by squashing the whole history of the vim-surround repository into a single one:

1bda0bd [3 minutes ago] (HEAD, stree) Merge commit 'ca1f4da9f0b93346bba9a430c889a95f75dc0a83' as '.vim/bundle/tpope-vim-surround' [Nicola Paolucci]
ca1f4da [3 minutes ago] Squashed '.vim/bundle/tpope-vim-surround/' content from commit 02199ea [Nicola Paolucci]

If after a while you want to update the code of the plugin from the upstream repository you can just subtree pull:

git subtree pull --prefix .vim/bundle/tpope-vim-surround https://bitbucket.org/vim-plugins-mirror/vim-surround.git master --squash

This is very quick and painless but the commands are slightly lengthy and hard to remember. We can make the commands shorter by adding the sub-project as a remote.

Adding the sub-project as a remote

Adding the subtree as a remote allows us to refer to it in shorter form:

git remote add -f tpope-vim-surround https://bitbucket.org/vim-plugins-mirror/vim-surround.git

Now we can add the subtree (as before), but now we can refer to the remote in short form:

git subtree add --prefix .vim/bundle/tpope-vim-surround tpope-vim-surround master --squash

The command to update the sub-project at a later date becomes:

git fetch tpope-vim-surround master
git subtree pull --prefix .vim/bundle/tpope-vim-surround tpope-vim-surround master --squash

Contributing back to upstream

We can freely commit our fixes to the sub-project in our local working directory now.

When it’s time to contribute back to the upstream project we need to fork the project and add it as another remote:

git remote add durdn-vim-surround ssh://git@bitbucket.org/durdn/vim-surround.git

Now we can use the subtree push command like the following:

git subtree push --prefix=.vim/bundle/tpope-vim-surround/ durdn-vim-surround master

git push using:  durdn-vim-surround master
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 308 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
To ssh://git@bitbucket.org/durdn/vim-surround.git
  02199ea..dcacd4b  dcacd4b21fe51c9b5824370b3b224c440b3470cb -} master

After this we’re ready and we can open a pull-request to the maintainer of the package.

Without using the subtree command

git subtree is different from the subtree merge strategy. You can still use the merge strategy even if for some reason git subtree is not available. Here is how you would go about it:

Add the dependency as a simple git remote:

git remote add -f tpope-vim-surround https://bitbucket.org/vim-plugins-mirror/vim-surround.git

Before reading the contents of the dependency into the repository it’s important to record a merge so that we can track the entire tree history of the plug-in up to this point:

git merge -s ours --no-commit tpope-vim-surround/master

Which outputs:

Automatic merge went well; stopped before committing as requested

We then read the content of the latest tree-object in the plugin repository into our working directory ready to be committed:

git read-tree --prefix=.vim/bundle/tpope-vim-surround/ -u tpope-vim-surround/master

Now we can commit (and it will be a merge commit that will preserve the history of the tree we read):

git ci -m"[subtree] adding tpope-vim-surround"

[stree 779b094] [subtree] adding tpope-vim-surround

When we want to update the project we can now pull using the subtree merge strategy:

git pull -s subtree tpope-vim-surround master

Conclusions

After having used submodule for a while I appreciate git subtree much more, lots of submodule problems are superseded and solved by subtree. As usual, with all things git, there is a learning curve to make the most of the feature.

Follow me @durdn for more Git rocking. And check out Atlassian Stash if you’re looking for a killer tool to manage your Git repos.

Update: I just wrote a new article on the power of Git subtree.

The classpath*:conf/appContext.xml simply means that all appContext.xml files under conf folders in all your jars on the classpath will be picked up and joined into one big application context.

In contrast, classpath:conf/appContext.xml will load only one such file… the first one found on your classpath.

One very important thing - if you use the * and Spring finds no matches, it will not complain. If you don’t use the * and there are no matches, the context will not start up!

What is the difference between “classpath:” and “classpath:/” in Spring XML?

I don’t see any difference between these two. The biggest difference that you will see is that the relative path and the * on the classpath location

Here is an excerpt from Spring Resources, look for section 4.7.2.2

Classpath*:

The classpath*: prefix can also be combined with a PathMatcher pattern in the rest of the location path, for example classpath*:META-INF/*-beans.xml. In this case, the resolution strategy is fairly simple: a ClassLoader.getResources() call is used on the last non-wildcard path segment to get all the matching resources in the class loader hierarchy, and then off each resource the same PathMatcher resoltion strategy described above is used for the wildcard subpath.

This means that a pattern like classpath*:*.xml will not retrieve files from the root of jar files but rather only from the root of expanded directories. This originates from a limitation in the JDK’s ClassLoader.getResources() method which only returns file system locations for a passed-in empty string (indicating potential roots to search).