create database localchatterprod default charset UTF8 default collate utf8-bin;
grant all on localchatterprod.* to 'localchatter'@'localhost' identified by 'localchatter';
grant all on localchatterprod.* to 'localchatter'@'localhost.localdomain' identified by 'localchatter';
grant all on localchatterprod.* to 'localchatter'@'%' identified by 'localchatter';




Overall workflow


-- Pre step for scraped resources --

Scraper does the scraping, and collates a deposit record
Scraper puts deposit record in local queue

Update service scans queue and for each changed record
Scraper contacts api webapp, authenticates and then deposits the record



API App
Gets incoming record and canonicalizes it
Puts canonicalised record into elasitcsearch index, no real need to store canonicalised record



# Testing

curl http://localhost:9200/_search?pretty=1&q=*
http://localhost:9200/localchatter/_search/?pretty=1&q=*

List of FIS departments:
http://findyourfis.daycaretrust.org.uk/kb5/findyourfis/results.page?qt=&region=


Info about FB login from http://developers.facebook.com
And about getting user info
http://stackoverflow.com/questions/15151717/grails-facebook-login-using-spring-security-how-to-know-logged-in-user-name

testing
http://localhost:8080/localchatter/admin/data/localchatter/sourcerecs/51581eece4b0546a8d7cd16f#.UWBd6aC8w7w



Kent sitemap

https://fisonline.tribalhosted.co.uk/Kent/Publications/Sitemap.xml



https://fisonline.tribalhosted.co.uk/TowerHamlets/EarlyYears/PublicEnquiry/Search.aspx?letter=A

Authority so far:

Kent:Publications
https://fisonline.tribalhosted.co.uk/Kent/EarlyYears/FSD/Search.aspx?letter=A

TowerHamlets:PublicEnquiry
https://fisonline.tribalhosted.co.uk/TowerHamlets/EarlyYears/PublicEnquiry/Search.aspx?letter=a

centralbedfordshire:fiso:publicenquiry
https://fisonline.tribalhosted.co.uk/centralbedfordshire/fiso/publicenquiry/Search.aspx?letter=a


Bracknell
https://fisonline.tribalhosted.co.uk/Bracknell/EarlyYears/PublicEnquiry/Search.aspx?letter=a
https://fisonline.tribalhosted.co.uk/Bracknell/EarlyYears/PublicEnquiry/Search.aspx?providerID=1486
https://fisonline.tribalhosted.co.uk/Bracknell/EarlyYears/PublicEnquiry/Print.aspx?ws=1&ln=1&id=pIKlifgzUFc%3D


Doncaster
https://fisonline.tribalhosted.co.uk/Doncaster/EarlyYears/FamilyServiceDirectory/Print.aspx?ws=2&ln=1&id=HpirRvTUPwc%3D


https://fisonline.tribalhosted.co.uk/${Authority}/[EarlyYears|]/${LocalCollectionName}/Search.aspx?letter=A

Details link:
https://fisonline.tribalhosted.co.uk/TowerHamlets/EarlyYears/PublicEnquiry/Search.aspx?providerID=313