Bug: duplicate key value violates unique constraint "PK_SMStreams" #12
Comments
|
I'm not too sure. In the newer release of StreamMaster (as in, after we have commit history for) the container ran a script called did_migrate_streams.shWhere I converted it to a SQL migration script, rather than having some kind of rogue shell script handling this DB specific logic. It should only run once. Where My guess based on your logs for: Could you try this SQL script on your DB, and see if it produces an outcome/collision? WITH generated_ids AS (
WITH temp_SMStreams AS (
SELECT "Id", "Url", "CUID", "ChannelId", "EPGID", "TVGName", "Name", "M3UFileId"
FROM "SMStreams"
WHERE "M3UFileId" IS NOT NULL AND "M3UFileId" >= 0
),
temp_M3UFiles AS (
SELECT "Id", COALESCE("M3UKey", 0) AS "M3UKey"
FROM "M3UFiles"
)
SELECT
s."Id" as old_id,
CASE f."M3UKey"
WHEN 0 THEN md5(concat(s."Url", '_', s."M3UFileId"))
WHEN 1 THEN md5(concat(s."CUID", '_', s."M3UFileId"))
WHEN 2 THEN md5(concat(s."ChannelId", '_', s."M3UFileId"))
WHEN 3 THEN md5(concat(s."EPGID", '_', s."M3UFileId"))
WHEN 4 THEN md5(concat(COALESCE(s."TVGName", s."Name"), '_', s."M3UFileId"))
WHEN 5 THEN CASE
WHEN s."TVGName" IS NOT NULL AND s."EPGID" IS NOT NULL
THEN md5(concat(s."TVGName" || '_' || s."EPGID", '_', s."M3UFileId"))
ELSE NULL
END
WHEN 6 THEN md5(concat(s."Name", '_', s."M3UFileId"))
WHEN 7 THEN CASE
WHEN s."Name" IS NOT NULL AND s."EPGID" IS NOT NULL
THEN md5(concat(s."Name" || '_' || s."EPGID", '_', s."M3UFileId"))
ELSE NULL
END
END as new_id,
s."M3UFileId" as m3ufileid,
f."M3UKey" as key_type,
s."Name" as channel_name
FROM temp_SMStreams s
LEFT JOIN temp_M3UFiles f ON s."M3UFileId" = f."Id"
)
SELECT
g1.old_id,
g1.new_id,
g1.m3ufileid,
g1.key_type,
g1.channel_name,
COUNT(*) OVER (PARTITION BY g1.new_id) as collision_count
FROM generated_ids g1
WHERE EXISTS (
SELECT 1
FROM generated_ids g2
WHERE g1.new_id = g2.new_id
AND g1.old_id != g2.old_id
)
ORDER BY g1.new_id, g1.m3ufileid; |
|
Just tried the script and it spits out tons of colisions, some counts up to 800+! I've trimmed the channel name slightlyto help with formatting |
|
Hmm, that does seem like a lot... The From your output, it seems md5(concat(COALESCE(s."TVGName", s."Name"), '_', s."M3UFileId"))Given that your I would just say, why not combine the private static string? GenerateM3UKeyValue(M3UKey m3uKey, SMStream smStream)
{
string? key = m3uKey switch
{
M3UKey.URL => smStream.Url,
M3UKey.CUID => smStream.CUID,
M3UKey.ChannelId => smStream.ChannelId,
M3UKey.TvgID => smStream.EPGID,
M3UKey.TvgName => string.IsNullOrEmpty(smStream.TVGName) ? smStream.Name : smStream.TVGName,
M3UKey.Name => smStream.Name,
M3UKey.TvgName_TvgID =>
(!string.IsNullOrEmpty(smStream.TVGName) && !string.IsNullOrEmpty(smStream.EPGID))
? $"{smStream.TVGName}_{smStream.EPGID}"
: null,
M3UKey.Name_TvgID =>
(!string.IsNullOrEmpty(smStream.Name) && !string.IsNullOrEmpty(smStream.EPGID))
? $"{smStream.Name}_{smStream.EPGID}"
: null,
_ => throw new ArgumentOutOfRangeException(nameof(m3uKey), m3uKey, null),
};
return string.IsNullOrEmpty(key) ? null : FileUtil.EncodeToMD5(key);
}So that can't be the way to fix it, or else the key the back-end generated would be misaligned causing further issues. Could change it in two places, though I am not then sure that some users with working set ups would then break without running some new ID generation migration. |
|
I see what you mean, it could be that it had been fixed in a newer update maybe. I'm happy to test and help fixing any issues that would arise from this change. This was one of the reasons I could not used the fork I mentioned, some streams were missing because the TVGName was missing from the EPG file so it would only load one stream. I'm not sure how you could do it another way, do you think that joining both as a single Id would be feasible? |
I agree that it makes sense that it should be able to handle such cases, ideally. I assume that If the M3U Key is used to locate a channel in an M3U. Then I don't know why you wouldn't just combine all of these values available per channel as a single identifier. It should then avoid these types of issues? I do see though, if you edit an M3U there is a "Channel Name Field" option here: I wonder if this is what drives the |
That's a really good point! You're probably right, because the dropdown weren't working for me at the beginning I completely forgot about those options. I will give that a go tomorrow to see if it helps when tuning the previous script. |
|
Ok so, I removed all the m3u files I had, then added 6 m3u different files for testing and a couple things happened:
Now I'm not sure what happened with 1, but I thinks this issue is fixed by picking the correct fields as Id, would be interesting if we could help choosing this, because apparently you won't noticed until it's too late and you're migrating, maybe a check that would just read something like, |
I guess that the "Add EPG" dialog could have some kind of "Test" button, or some kind of verification that runs automatically before the "accept" button becomes enabled. The issue mostly, is that the M3U needs to exist in some way, to be able try and parse its contents to validate if a collision would happen. Though even the "M3u key Mapping" should probably have options disabled if they don't exist in the M3U too. |
|
It's a fair point, this might be a lot more work than it looks |
|
Hmm, wonder why it's not letting me convert it to a discussion. Will try again later... Otherwise I think it would be good to move this to a discussion under "Ideas" as a feature request. Where more upfront validation is performed before committing to the initial save. |
|
What is the solution for this? I tried the migration procedure and I'm running in to this. As I mentioned earlier in the duplicate bug, I could get errors to go away by simply dropping the streammaster db and letting the scripts rebuild, then restoring the various configuration files in the db, but that feel like a poor hack and may be the source of other issues I'm running in to such as visibilty of stream groups not working right. |
|
If you have a way to reproduce going from state A -> B in terms of some kind of upgrade causing problems, then I am eager to know. Starting fresh really isn't ideal in my eyes. Could you add a bit about which version you were maybe running before, and what migration steps you have taken? It could be that this fork is attempting to do a previous migration again which is causing the conflict. Though as I wrote earlier in this issue discussion, it should be avoided so long as |
Describe the Bug
I've noticed this a few times, I just updated the container to use the 0.7.3 version, and got this error and cannot start up. I has happened in the past, not necessarely when updating but when restarting it.
Could this be because a migration is not compatible with 0.3.1 (the one I was on), or was there duplicate data, and if so, how was it allowed?
Stream Master Version
0.7.3
Relevant Logs
The text was updated successfully, but these errors were encountered: