Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/126 limit api importer #128

Closed
wants to merge 5 commits into from
Closed

Conversation

tamcap
Copy link
Contributor

@tamcap tamcap commented May 16, 2014

This fixes #126.

It should be only merged 24 hours after #127 is merged, as I would like for a complete import to happen first (that's how I tested it). In theory it should work on any database state... but that's untested.

This is still not optimal - the inspections import has a for loop that causes a few hundreds calls to the Durham County API - most of them are unnecessary. Ideas?

@tamcap tamcap closed this May 16, 2014
@tamcap
Copy link
Contributor Author

tamcap commented May 16, 2014

New, better commits. Cleaner code. Please review.

@tamcap tamcap reopened this May 16, 2014
@copelco
Copy link
Member

copelco commented May 20, 2014

I think this branch should be updated with the latest from develop

@tamcap
Copy link
Contributor Author

tamcap commented May 21, 2014

Rebased to current + pep8 fixes.

'schedule': crontab(minute=0, hour=0, day_of_week='sun'), # Execute every sunday at midnight
},
'import-durham-data-incr': {
'task': 'eatsmart.locations.durham.tasks.import_durham_data_incr',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this task is not defined

tamcap added 4 commits June 4, 2014 14:59
* Much faster imports from Durham County due to incremental nature of the data
* Rewrote some of the API code for better clarity
@tamcap
Copy link
Contributor Author

tamcap commented Jun 4, 2014

New changes as discussed with @victor-rocha during the 6/3/2014 meeting.

Refactoring some code
@copelco copelco added this to the Phase 3 milestone Jul 1, 2014
@tamcap tamcap closed this May 11, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make API importer only update new records
3 participants