The Google Webmaster team has posted a blog entry that should help to shed some light on how Google crawls and indexes sites. The entry is supplemented by a thirteen slide presentation entitled Optimize your URLs[:] Best practices for crawling & indexing.

The presentation uses real world examples. Names have been changed to protect the guilty! Recommendations are varied and include removing user-specific details from URLs (such as session ID's), and assigning one unique set of content to each URL.

Want to learn more?

Official Google Webmaster Central Blog: Optimize your crawling & indexing