Sync stops on mozilla separator, detecting it as invalid or corrupt bookmark, after restored with 1.4 even if resync on beta [v1.5.0beta]

[Poikilos]

Steps to reproduce

I have a huge Firefox bookmark library many years old with many strangely named links made in various operating systems and Firefox versions (attached under step 2). These bookmarks sync correctly with Firefox Sync. For some reason certain protocols aren't accepted, so pdf bookmarks aren't (since they use chrome-extension:// protocol).

1. Install xBrowserSync
2. Connect to account
3. Restore this JSON backup (using the standard firefox Restore feature via Ctrl+Shift+o or in menu, Library, Bookmarks, Show All Bookmarks [at bottom])
   bookmarks-2019-04-04.json.tar.gz

Expected behaviour

• Allow other protocols
  • automatically change protocol when needed-- chrome-extension://oemmndcbldboiebfnladdacbdfmadadm/http://www.belkin.com/support/dl/Using_F5D7230_Router_As_Access_Point.pdf would become http://www.belkin.com/support/dl/Using_F5D7230_Router_As_Access_Point.pdf)
  • somehow save other bookmarks instead of failing--maybe just allow any protocol (people may want a link to about:config and other things like that, and other extension links not like the one above)
• Fill in missing required bookmark fields with sequential (unused number), random (GUID/other) or speculative data (https://www.github.com/xbrowsersync gets titled github or github.com or www.github.com/xbrowsersync to avoid overlap), instead of failing with missing field error and losing the bookmark.
  • :edit: I've created a program and confirmed that the only title fields that are blank in my JSON file are root and x-moz-place-separator. Blank title fields should remain blank in those cases. x-moz-place-separator and root entries with blank titles should also be preserved by xBrowserSync despite not having a uri field with http protocol. If uploaded via the API, downloading them with chrome should be tested--the Chrome extension should automatically ignore separators since Chrome doesn't support any type of separator.
    Separator example:

                {
                  "guid": "nqM8HkdjX6CS",
                  "title": "",
                  "index": 0,
                  "dateAdded": 1237434283343000,
                  "lastModified": 1237434283343000,
                  "id": 4918,
                  "typeCode": 3,
                  "type": "text/x-moz-place-separator"
                },

The root entry:

{
  "guid": "root________",
  "title": "",
  "index": 0,
  "dateAdded": 1529440240154000,
  "lastModified": 1554318649423000,
  "id": 1,
  "typeCode": 2,
  "type": "text/x-moz-place-container",
  "root": "placesRoot",
  "children": [
    {
. . .

Actual behaviour

• Sync stops (in 1.4, xBrowserSync logo stays gray--in 1.5.0beta5, icon has red x--1.5.0beta5).
  • after one bookmark has missing field: bookmark.title
• Next time you click xBrowserSync, it asks to sign in again (but remembers sync id). The loop continues forever.
• Even after removing the addon and re-adding it, the problem stays permanently. This may indicate the data on the server is now faulty, and that neither the server nor client can correct it.
  • only the 3 bookmarks which I manually added as a test (before restoring) are actually downloaded: https://www.youtube.com in Other, and https://github.com in Bookmarks toolbar, and the brave/brave-browser/issues link in projects folder of bookmarks toolbar

Log:

20190404110157216	[trace]	Installed v1.5.0beta5
20190404110157219	[trace]	Starting up
20190404110157226	[trace]	{"appVersion":"1.5.0beta5","platform":{"name":"firefox","version":"66.0.0","mobile":false,"os":"Fedora"}}
20190404110220645	[trace]	Sync x5tdx4 (pull) queued (1 waiting to sync)
20190404110220807	[trace]	Error populating bookmarks menu.
20190404110220811	[error]	bookmarks.sync: bookmark.title is undefined	isSeparator@moz-extension://42a87aa2-b1f5-46b3-9ca4-68d5d9fee4b6/js/background.min.js:2498:50 createLocalBookmarksFromXBookmarks/</<@moz-extension://42a87aa2-b1f5-46b3-9ca4-68d5d9fee4b6/js/background.min.js:3980:24 k/<@moz-extension://42a87aa2-b1f5-46b3-9ca4-68d5d9fee4b6/js/background.min.js:140:44 $digest@moz-extension://42a87aa2-b1f5-46b3-9ca4-68d5d9fee4b6/js/background.min.js:151:185 $evalAsync/<@moz-extension://42a87aa2-b1f5-46b3-9ca4-68d5d9fee4b6/js/background.min.js:154:324 Tg/this.completeTask@moz-extension://42a87aa2-b1f5-46b3-9ca4-68d5d9fee4b6/js/background.min.js:166:15 Ag/h.defer/d<@moz-extension://42a87aa2-b1f5-46b3-9ca4-68d5d9fee4b6/js/background.min.js:52:1 

Initial Log

(first time install of 1.5.0beta5)

20190329111606172	[trace]	Installed v1.5.0beta5
20190329111606181	[trace]	Starting up
20190329111606200	[trace]	{"appVersion":"1.5.0beta5","platform":{"name":"firefox","version":"60.0.0","mobile":false,"os":"Linux x86_64"}}
20190329111633769	[trace]	Starting up
20190329111634472	[trace]	{"displayIntro":false,"networkDisconnected":false,"appVersion":"1.5.0beta5","platform":{"name":"firefox","version":"60.0.0","mobile":false,"os":"Linux x86_64"}}
20190329111657074	[trace]	Sync ndp6hg (pull) queued (1 waiting to sync)
20190329111703128	[trace]	Error populating bookmarks toolbar.
20190329111703138	[error]	bookmarks.sync: result.dateAdded is undefined
20190329111731658	[trace]	Bookmark url unsupported: chrome-extension://oemmndcbldboiebfnladdacbdfmadadm/http://www.belkin.com/support/dl/Using_F5D7230_Router_As_Access_Point.pdf
20190329111746328	[trace]	Bookmark url unsupported: chrome-extension://oemmndcbldboiebfnladdacbdfmadadm/http://dialnet.unirioja.es/descarga/articulo/1091297.pdf
20190329111750105	[trace]	Bookmark url unsupported: chrome-extension://oemmndcbldboiebfnladdacbdfmadadm/http://libertad.org/wp-content/uploads/2014/01/Puntos-destacados-del-Indice-2014-de-Libertad-Economica.pdf
20190329111848452	[trace]	Sync o7dopf (pull) queued (1 waiting to sync)
20190329111852812	[trace]	Error populating bookmarks toolbar.
20190329111852823	[error]	bookmarks.sync: result.dateAdded is undefined
20190329111923019	[trace]	Bookmark url unsupported: chrome-extension://oemmndcbldboiebfnladdacbdfmadadm/http://www.belkin.com/support/dl/Using_F5D7230_Router_As_Access_Point.pdf

1.4 Log

INFO: createLocalBookmark: {"index":1,"parentId":"toolbar_____","title":"GitHub","url":"https://github.com/"}
    INFO: Sync data: {"type":2,"command":1,"deferred":{"promise":{"$$state":{"status":2,"value":{},"processScheduled":false,"pur":true}}}}
    ERROR: Error: Invalid bookmark: {"title":"","type":1,"url":"data:","index":1,"parentGuid":"menu________"}
    INFO: createLocalBookmark: {"index":1,"parentId":"menu________","url":"data:"}
    INFO: createLocalBookmark: {"index":0,"parentId":"pimyLH_tfk9A","title":"Bold Styling for GitHub Markdown Does not Work · Issue #3991 · brave/brave-browser","url":"https://github.com/brave/brave-browser/issues/3991"}
    INFO: createLocalBookmark: {"index":0,"parentId":"menu________","title":"Recently Bookmarked","url":"place:parent=menu________&parent=unfiled_____&parent=toolbar_____&queryType=1&sort=12&maxResults=10&excludeQueries=1"}
    INFO: createLocalBookmark: {"index":0,"parentId":"toolbar_____","title":"projects"}
    INFO: createLocalBookmark: {"index":0,"parentId":"unfiled_____","title":"YouTube","url":"https://www.youtube.com/"}

Exporting from xBookmarkSync contains only the following:

{
   "xBrowserSync":{
      "id":"57cd3b47066d454daff9bf2cea1ba7bc",
      "bookmarks":[
         {
            "title":"[xbs] Menu",
            "children":[
               {
                  "title":"Recently Bookmarked",
                  "url":"place:parent=menu________&parent=unfiled_____&parent=toolbar_____&queryType=1&sort=12&maxResults=10&excludeQueries=1",
                  "id":4
               },
               {
                  "url":"data:",
                  "id":2
               },
               {
                  "title":"1.Sites, Downloads",
                  "children":[
                     {
                        "title":"Openwares.org :: Software is Free",
                        "url":"http://www.openwares.org/",
                        "id":5
                     },
                     {
                        "title":"Openwares.org :: Software is Free",
                        "url":"http://www.openwares.org/index.php?option=content&task=view&id=13&Itemid=31",
                        "id":7
                     }
                  ],
                  "id":3
               },
               {
                  "title":"Frequently Visited",
                  "children":[

                  ],
                  "id":6
               }
            ],
            "id":1
         }
      ]
   }
}

Client app information

Firefox 66.0.2 64-bit

Other Information

I'm not sure whether this is related to #75

[Poikilos]

@nero120

[nero120]

@Poikilos sorry for not responding sooner, literally not had any free time for the last couple of weeks. Thank you very much for the detailed report, I'll dig into this as soon as I can.

[nero120]

@Poikilos apologies for the delay, I've fixed a bug that was causing the bookmarks.sync: bookmark.title is undefined error in e1dcb9a. Also, a failed sync should not longer corrupt synced data on the server.

Please could you download v1.5.0beta6 and give it a try?

I'm interested to see if you still get the bookmarks.sync: result.dateAdded is undefined error as this I cannot replicate. A user that was running ArchLinux had reported this also, what distro are you running this on?

Allow other protocols

xBrowserSync does not restrict bookmarks based on protocol, the browser API is what restricts you. For example, you cannot create a bookmark in Firefox (even using the UI) with a chrome:// protocol, it will error, xbs converts these bookmark urls to about:newtab so that the bookmark is still synced (albeit with a url that goes nowhere). However if you try creating a bookmark with url test://thisisatest.com it will succeed in both Chrome and Firefox.

(Also, thank you very much for the very detailed information in the issue, if only everyone who raised issues were like you!! 🙇)

[Poikilos]

@nero120, thanks for getting back to me. I successfully installed it on Brave (Chrome-based, Version 0.62.51 Chromium: 73.0.3683.103 (Official Build) (64-bit)), but I am not able to get the beta working on Firefox 66.0.3. The plugin installs, but has a sync error--"An error occurred when trying to create a local bookmark" according to the UI. I can't see how having it installed on Brave could have caused the issue because I only have 2 bookmarks in Brave:

projects
  - [https://github.com/brave/brave-browser/issues/3991](Bold Styling for GitHub Markdown Does not Work · Issue #3991 · brave/brave-browser)
[https://github.com](GitHub)

Log:

20190502060915227	[trace]	Installed v1.5.0beta6
20190502060915231	[trace]	Starting up
20190502060915241	[trace]	{"appVersion":"1.5.0beta6","platform":{"name":"firefox","version":"66.0.0","mobile":false,"os":"Fedora"}}
20190502061010407	[trace]	Starting up
20190502061010483	[trace]	{"appVersion":"1.5.0beta6","platform":{"name":"firefox","version":"66.0.0","mobile":false,"os":"Fedora"}}
20190502061122835	[trace]	Sync saiqat (pull) queued (1 waiting to sync)
20190502061123058	[trace]	Failed to create local bookmark: {"parentId":"5Cv_tdN0jxl7","url":"data:"}
20190502061123066	[trace]	Error populating other bookmarks.
20190502061123071	[error]	bookmarks.sync: [10103] FailedCreateLocalBookmarks
20190502061202049	[trace]	Sync pfj2n3 (pull) queued (1 waiting to sync)
20190502061202239	[trace]	Failed to create local bookmark: {"parentId":"bORZdYYlErvZ","url":"data:"}
20190502061202255	[error]	bookmarks.sync: [10103] FailedCreateLocalBookmarks

Additionally, I had to customize the toolbar and manually drag the xBrowserSync icon onto it. I'm not sure whether that is normal.

[nero120]

I successfully installed it on Brave

@Poikilos FYI xBrowserSync is not currently supported on Brave, only (official) Chrome and Firefox are supported at this time. It may run but it depends on how closely Brave implements Chome's extension APIs and bookmark functionality. If you'd like to request support for Brave be added please submit a separate feature request.

The plugin installs, but has a sync error--"An error occurred when trying to create a local bookmark"

It appears you are attempting to sync a bookmark with a url of data:. This could be something specific to Brave, perhaps not displayed in the UI so you wouldn't know about it (Vivaldi also has similar quirks).

If you create a feature request for Brave I'll look at implementing support once support for Opera and Vivaldi have been added.

[Poikilos]

I see, I can confirm that the issue stated in the title is resolved. Sync works between Chrome and Firefox!