Fix race conditions while running migrations for fresh installs

README.md

@@ -6,9 +6,17 @@
 
 We're open for [contributions](./CONTRIBUTING.md)!
 
+## Important notice when upgrading from any release older than [`2.024.19`](https://github.com/kobotoolbox/kpi/releases/tag/2.024.19)
+
+Prior to release [`2.024.19`](https://github.com/kobotoolbox/kpi/releases/tag/2.024.19), this project (KPI) and [KoboCAT](https://github.com/kobotoolbox/kobocat) were two separated projects.
+KoboCAT is now part of KPI code base and its repository has been archived.
+
+KoboCAT deprecation notices will be maintained in this repository.
+[More details here](./kobo/apps/openrosa/README.md)
+
 ## Important notice when upgrading from any release older than [`2.020.18`](https://github.com/kobotoolbox/kpi/releases/tag/2.020.18)
 
-Prior to release [`2.020.18`](https://github.com/kobotoolbox/kpi/releases/tag/2.020.18), this project (KPI) and [KoBoCAT](https://github.com/kobotoolbox/kobocat) both shared a common Postgres database. They now each have their own. **If you are upgrading an existing single-database installation, you must follow [these instructions](https://community.kobotoolbox.org/t/upgrading-to-separate-databases-for-kpi-and-kobocat/7202)** to migrate the KPI tables to a new database and adjust your configuration appropriately.
+Prior to release [`2.020.18`](https://github.com/kobotoolbox/kpi/releases/tag/2.020.18), this project (KPI) and [KoboCAT](https://github.com/kobotoolbox/kobocat) both shared a common Postgres database. They now each have their own. **If you are upgrading an existing single-database installation, you must follow [these instructions](https://community.kobotoolbox.org/t/upgrading-to-separate-databases-for-kpi-and-kobocat/7202)** to migrate the KPI tables to a new database and adjust your configuration appropriately.
 
 If you do not want to upgrade at this time, please use the [`shared-database-obsolete`](https://github.com/kobotoolbox/kpi/tree/shared-database-obsolete) branch instead.
 
@@ -35,7 +43,7 @@ syntax, see the documentation at the top of
 
 ## Admin reports
 
-There are several types of data reports available to superusers. 
+There are several types of data reports available to superusers.
 * Full list of users including their details provided during signup, number of deployed projects (XForm count), number of submissions, date joined, and last login: `<kpi base url>/superuser_stats/user_report/`. File being created is a CSV, so don't download immediately to wait for server to be finished writing to the file (it will download even if incomplete).
 * Monthly aggregate figures for number of forms, deployed projects, and submissions (from kobocat): `<kc server domain>/<superuser username>/superuser_stats/`
 

hub/admin/password_validation.py

@@ -3,7 +3,7 @@
 from django.db import transaction
 from django.utils.html import format_html
 
-from kpi.deployment_backends.kc_access.shadow_models import KobocatUserProfile
+from kobo.apps.openrosa.apps.main.models import UserProfile
 from .filters import PasswordValidationAdvancedSearchFilter
 from .mixins import AdvancedSearchMixin
 from ..models import ExtraUserDetail
@@ -67,12 +67,17 @@ def get_queryset(self, request):
 
     @admin.display(description='Validated')
     def get_validated_password(self, obj):
-        value = True
+        value = False
         try:
             value = obj.extra_details.validated_password
         except obj.extra_details.RelatedObjectDoesNotExist:
             pass
 
+        try:
+            value = value and obj.profile.validated_password
+        except obj.profile.RelatedObjectDoesNotExist:
+            pass
+
         return format_html(
             '<img src="/static/admin/img/icon-{}.svg" alt="{}">',
             'yes' if value else 'no',
@@ -88,8 +93,14 @@ def invalidate_passwords(self, request, queryset, **kwargs):
             ExtraUserDetail.objects.filter(user_id__in=user_ids).update(
                 validated_password=False
             )
-            KobocatUserProfile.objects.filter(user_id__in=user_ids).update(
-                validated_password=False
+            UserProfile.objects.bulk_create(
+                [
+                    UserProfile(user_id=user_id, validated_password=False)
+                    for user_id in user_ids
+                ],
+                update_conflicts=True,
+                unique_fields=['user_id'],
+                update_fields=['validated_password'],
             )
 
         self.message_user(
@@ -107,8 +118,14 @@ def validate_passwords(self, request, queryset, **kwargs):
             ExtraUserDetail.objects.filter(user_id__in=user_ids).update(
                 validated_password=True
             )
-            KobocatUserProfile.objects.filter(user_id__in=user_ids).update(
-                validated_password=True
+            UserProfile.objects.bulk_create(
+                [
+                    UserProfile(user_id=user_id, validated_password=True)
+                    for user_id in user_ids
+                ],
+                update_conflicts=True,
+                unique_fields=['user_id'],
+                update_fields=['validated_password'],
             )
 
         self.message_user(

jsapp/js/components/formGallery/formGallery.component.tsx

@@ -167,7 +167,7 @@ export default function FormGallery(props: FormGalleryProps) {
               target='_blank'
             >
               <img
-                src={attachment.download_url}
+                src={attachment.download_small_url}
                 alt={attachment.filename}
                 width='150'
                 loading='lazy'

kobo/apps/kobo_auth/models.py

@@ -17,11 +17,11 @@ class Meta:
         swappable = 'AUTH_USER_MODEL'
 
     def has_perm(self, perm, obj=None):
-        # If it is a Kobocat permissions, check permission in Kobocat DB first
+        # If it is a KoboCAT permissions, check permission in KoboCAT DB first
         # 3 options:
-        # - `obj` is not None and its app_label belongs to Kobocat
+        # - `obj` is not None and its app_label belongs to KoboCAT
         # - `perm` format is <app_label>.<perm>, we check the app label
-        # - `perm` belongs to Kobocat permission codenames
+        # - `perm` belongs to KoboCAT permission codenames
         if obj:
             if obj._meta.app_label in OPENROSA_APP_LABELS:
                 with use_db(settings.OPENROSA_DB_ALIAS):

kobo/apps/openrosa/README.md

@@ -0,0 +1,115 @@
+# KoboCAT
+
+## Important notice when upgrading from any release older than [`2.020.18`](https://github.com/kobotoolbox/kobocat/releases/tag/2.020.18)
+
+Up to and including release [`2.020.18`](https://github.com/kobotoolbox/kobocat/releases/tag/2.020.18), this project (KoboCAT) and [KPI](https://github.com/kobotoolbox/kpi) both shared a common Postgres database. They now each have their own. **If you are upgrading an existing single-database installation, you must follow [these instructions](https://community.kobotoolbox.org/t/upgrading-to-separate-databases-for-kpi-and-kobocat/7202)** to migrate the KPI tables to a new database and adjust your configuration appropriately.
+
+If you do not want to upgrade at this time, please use the [`shared-database-obsolete`](https://github.com/kobotoolbox/kobocat/tree/shared-database-obsolete) branch instead.
+
+## Deprecation Notices
+
+Much of the user-facing features of this application are being migrated
+to <https://github.com/kobotoolbox/kpi>. KoboCAT's data-access API and
+OpenRosa functions will remain intact, and any plans to the contrary
+will be announced well in advance. For more details and discussion,
+please refer to
+<https://community.kobotoolbox.org/t/contemplating-the-future-of-kobocat/2743>.
+
+As features are migrated, we will list them here along with the last
+release where each was present:
+  - Starting from version [2.024.19](https://github.com/kobotoolbox/kobocat/releases/tag/2.024.19),
+    the ability to import submissions in CSV to KoboCAT was
+    removed (i.e: `https://kobocat.domain.tld/api/v1/forms/<form_id>/csv_import`)
+    .
+  - On 14 June 2021, the ability to upload forms directly to KoboCAT was
+    removed, and it was announced that the legacy KoboCAT user interface would
+    be preserved for "a few more months". After more than two years, we have
+    removed the user interface and related endpoints entirely in release
+    [2.023.37](https://github.com/kobotoolbox/kobocat/releases/tag/2.023.37).
+    **This includes the ability to upload XLSForms via the legacy KoboCAT API.**
+    Please use the KPI `v2` API for all form management. Other removed features
+    should already be available in KPI as well. Please see
+    [REMOVALS.md](REMOVALS.md) for a complete list.
+  - To ensure security and stability, many endpoints that were already
+    available in KPI, long-unsupported, or underutilized have been removed in
+    release
+    [2.020.40](https://github.com/kobotoolbox/kobocat/releases/tag/2.020.40).
+    These were related to charts and stats, form cloning, form sharing, user
+    profiles, organizations / projects / teams, bamboo, and ziggy. For a full
+    list, please see [REMOVALS.md](REMOVALS.md). These endpoints were last
+    available in the release
+    [2.020.39](https://github.com/kobotoolbox/kobocat/releases/tag/2.020.39).
+  - REST Services - an improved version [has been added to
+    KPI](https://github.com/kobotoolbox/kpi/pull/1864). The last KoboCAT
+    release to contain legacy REST services is
+    [2.019.39](https://github.com/kobotoolbox/kobocat/releases/tag/2.019.39).
+
+## About
+
+kobocat is the data collection platform used in KoboToolbox. It is based
+on the excellent [onadata](http://github.com/onaio/onadata) platform
+developed by Ona LLC, which in itself is a redevelopment of the
+[formhub](http://github.com/SEL-Columbia/formhub) platform developed by
+the Sustainable Engineering Lab at Columbia University.
+
+Please refer to
+[kobo-install](https://github.com/kobotoolbox/kobo-install) for
+instructions on how to install KoboToolbox.
+
+## Code Structure
+
+  - **logger** - This app serves XForms to and receives submissions from
+    ODK Collect and Enketo.
+  - **viewer** - This app provides a csv and xls export of the data
+    stored in logger. This app uses a data dictionary as produced by
+    pyxform. It also provides a map and single survey view.
+  - **main** - This app is the glue that brings logger and viewer
+    together.
+
+## Localization
+
+To generate a locale from scratch (ex. Spanish)
+
+``` sh
+$ django-admin makemessages -l es -e py,html,email,txt ;
+$ for app in {main,viewer} ; do cd kobocat/apps/${app} && django-admin makemessages -d djangojs -l es && cd - ; done
+```
+
+To update PO files
+
+``` sh
+$ django-admin makemessages -a ;
+$ for app in {main,viewer} ; do cd kobocat/apps/${app} && django-admin makemessages -d djangojs -a && cd - ; done
+```
+
+To compile MO files and update live translations
+
+``` sh
+$ django-admin compilemessages ;
+$ for app in {main,viewer} ; do cd kobocat/apps/${app} && django-admin compilemessages && cd - ; done
+```
+## Testing in KoboCAT
+
+For kobo-install users, enter the folder for kobo-install and run this command
+
+```
+./run.py -cf exec kobocat bash
+```
+
+For all other users, enter the container using this command
+
+``` sh
+$ docker exec -it {{kobocat container}} /bin/bash
+```
+
+Run pip install the development dependencies
+
+``` sh
+$ pip install -r dependencies/pip/dev.txt
+```
+
+Run pytest to run all automated tests
+
+``` sh
+$ pytest
+```