Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Idea: Feature to aggregate transactions per month #252

Closed
dell-mic opened this issue Nov 18, 2020 · 22 comments · Fixed by #253, #254 or #256
Closed

Idea: Feature to aggregate transactions per month #252

dell-mic opened this issue Nov 18, 2020 · 22 comments · Fixed by #253, #254 or #256

Comments

@dell-mic
Copy link
Contributor

dell-mic commented Nov 18, 2020 •
edited
Loading

What?
I could imagine an optional parameter like --aggregate=month which should sum up transactions per month and booking type. When not given: continue with the current behavior.

Why?
Using auto invest on Mintos I have tens of thousands single transactions in PP already, and they are rapidly growing. I do not care about keeping books on each transaction, but rather overall performance of the P2P portfolio. IMHO interest payments per month are good enough for this purpose.

Am I missing something?
@ChrisRBe ChrisRBe mentioned this issue Nov 20, 2020
Merged
@ChrisRBe
Copy link
Owner

Hi @dell-mic,
Great idea. I will try to work on this this weekend. Starting out with some small clean up activities. Hope to proceed a bit further tomorrow.

@dell-mic
Copy link
Contributor Author

Cool! Let me know if there is anything I can help with.
(Having, however, very limited python skills)

@ChrisRBe
Copy link
Owner

ChrisRBe commented Nov 21, 2020 via email

@dell-mic
Copy link
Contributor Author

Sure thing. Can you describe what exactly you have in mind?

Is it enough to extend the current test file such that there are some transactions spanning several month?
Something like this?
image

@ChrisRBe
Copy link
Owner

ChrisRBe commented Nov 21, 2020 via email

@dell-mic
Copy link
Contributor Author

Sure thing it is technically a second file. What I meant is: I just created it by duplicating and editing the existing one ;)

See here: https://github.com/dell-mic/PP-P2P-Parser/blob/master/src/test/testdata/mintos_several_months.csv

@ChrisRBe
Copy link
Owner

ChrisRBe commented Nov 21, 2020 via email

@ChrisRBe
Copy link
Owner

ChrisRBe commented Nov 21, 2020 •
edited
Loading

Sure thing it is technically a second file. What I meant is: I just created it by duplicating and editing the existing one ;)

See here: https://github.com/dell-mic/PP-P2P-Parser/blob/master/src/test/testdata/mintos_several_months.csv

This looks good to me. Could you create a PR to merge into the feature branch?

@ChrisRBe
Copy link
Owner

Hi @dell-mic,
Could you have a look? I merged the PR now.. I hope it's working fine.

@ChrisRBe
Copy link
Owner

On an unrelated note... Do you have by any chance data for secondary market transactions or fees? Would be great if I could add it to one of the config files to check if they are processed properly.

@dell-mic
Copy link
Contributor Author

Cool, thank you very much for the fast implementation!
I will have a look tomorrow.

I should also have some examples for secondary market transactions in my Minto statement file which I can provide.

@dell-mic
Copy link
Contributor Author

I just gave it a try running against my Mintos statement file from 2017 until now: Results look very promising :)

I did run in a small glitch first: ModuleNotFoundError: No module named 'yaml' and fixed that by running pip3 install -r requirements.txt. Not sure if this is the best way to do this in the python world? (I am on macOS)

After that it seemed to work for me. There is a small difference between the interested amount PP is reporting vs. what Mintos says: In my case 2937.99 vs. 2894.71 which I assume could be rooted in rounding errors?

@dell-mic
Copy link
Contributor Author

Regarding the secondary market transactions. Here are some examples from my Mintos statement file. Mintos seems to have changed the file syntax / column ordering a bit. Should I adapt those to the current CSV formatting used in the test files?

"2018-09-16 15:35:45",435956010,"Loan 1817871-01 - secondary market transaction  ",-0.28,58.251478555098,EUR
"2018-09-16 15:35:45",435956011,"Loan 1817871-01 - discount/premium for secondary market transaction 435956010.",0.00056,58.252038555098,EUR
"2018-09-16 15:35:45",435956015,"Loan 4123052-01 - secondary market transaction  ",-3.3585154161915,54.893523138907,EUR
"2018-09-16 15:35:45",435956016,"Loan 4123052-01 - discount/premium for secondary market transaction 435956015.",0.0067170308323829,54.900240169739,EUR
"2018-09-16 15:35:45",435956020,"Loan 4129681-01 - secondary market transaction  ",-2.33,52.570240169739,EUR

@ChrisRBe
Copy link
Owner

Regarding the secondary market transactions. Here are some examples from my Mintos statement file. Mintos seems to have changed the file syntax / column ordering a bit. Should I adapt those to the current CSV formatting used in the test files?

"2018-09-16 15:35:45",435956010,"Loan 1817871-01 - secondary market transaction  ",-0.28,58.251478555098,EUR
"2018-09-16 15:35:45",435956011,"Loan 1817871-01 - discount/premium for secondary market transaction 435956010.",0.00056,58.252038555098,EUR
"2018-09-16 15:35:45",435956015,"Loan 4123052-01 - secondary market transaction  ",-3.3585154161915,54.893523138907,EUR
"2018-09-16 15:35:45",435956016,"Loan 4123052-01 - discount/premium for secondary market transaction 435956015.",0.0067170308323829,54.900240169739,EUR
"2018-09-16 15:35:45",435956020,"Loan 4129681-01 - secondary market transaction  ",-2.33,52.570240169739,EUR

Do you have an idea what kind of statements these are?

@ChrisRBe
Copy link
Owner

I just gave it a try running against my Mintos statement file from 2017 until now: Results look very promising :)

I did run in a small glitch first: ModuleNotFoundError: No module named 'yaml' and fixed that by running pip3 install -r requirements.txt. Not sure if this is the best way to do this in the python world? (I am on macOS)

After that it seemed to work for me. There is a small difference between the interested amount PP is reporting vs. what Mintos says: In my case 2937.99 vs. 2894.71 which I assume could be rooted in rounding errors?

It might be rounding errors, or some statements are not correctly recognized/processed by the script (not getting matched with the current regexes)

@dell-mic
Copy link
Contributor Author

Regarding the secondary market transactions. Here are some examples from my Mintos statement file. Mintos seems to have changed the file syntax / column ordering a bit. Should I adapt those to the current CSV formatting used in the test files?

"2018-09-16 15:35:45",435956010,"Loan 1817871-01 - secondary market transaction  ",-0.28,58.251478555098,EUR
"2018-09-16 15:35:45",435956011,"Loan 1817871-01 - discount/premium for secondary market transaction 435956010.",0.00056,58.252038555098,EUR
"2018-09-16 15:35:45",435956015,"Loan 4123052-01 - secondary market transaction  ",-3.3585154161915,54.893523138907,EUR
"2018-09-16 15:35:45",435956016,"Loan 4123052-01 - discount/premium for secondary market transaction 435956015.",0.0067170308323829,54.900240169739,EUR
"2018-09-16 15:35:45",435956020,"Loan 4129681-01 - secondary market transaction  ",-2.33,52.570240169739,EUR

Do you have an idea what kind of statements these are?

From what I understand Mintos reports those discounts you get when e.g. purchasing a loan below it's outstanding principal explicitly, instead of increasing the the effective interested rate.
IMHO from an investor perspective those could be treated just like received interest. (Which would be negative in case you paid a premium on the secondary market)

@dell-mic
Copy link
Contributor Author

A further note on the difference of the sum of interested received in PP vs. Mintos:
I just realized, that PP will not take transactions into account with a future date (at least not when using one of the default time filters like "5 Jahre"). Thus, the interested for November was not taken into account. When I change this date manually in PP it goes up to 2941.79 for me.

@ChrisRBe
Copy link
Owner

So now it's a bit more than what Mintos is reporting? And yes. Okay... that makes sense ... I did not take that into consideration uff.

@dell-mic
Copy link
Contributor Author

Yes, but the gap is less than before ;)

For me personally I would not mind this behavior, but I guess it might confuse others, if not documented. If you plan to fix it, I guess you would have to make sure to use a max date of today for the current month when aggregating.

@ChrisRBe
Copy link
Owner

I think I will leave it as is. Any potential fix is going to introduce additional headache:

  • If we're in the middle of the current month
  • PP will not import future dates
  • setting monthly date to today (to get it imported in PP)
  • import date of payments received will be today

Starting the script again next month will now correctly aggregate all payments received in the previous month for the last day of the month. Importing now into PP will create an additional entry for the previous one:

  • entry 1 with "today" date and money received until that day
  • entry 2 with "end of month" date and money received until "today" plus rest of month.
    This will basically report twice the amount received until "today".

The same behaviour would also apply if I were to use first day of month instead of last day of month. You would just have multiple bookings for the first of the month.

What I will do ... Add a statement in the output explaining that dates in the future can't be processed by PP.

@dell-mic
Copy link
Contributor Author

Good point, I haven't thought about PP importing the latest month twice.
As said before: I am completely fine with the current implementation. Definitely serves my purpose.
IMHO documenting the behavior for future users is a great idea.

Thanks again for the fast implementation!

@ChrisRBe ChrisRBe mentioned this issue Nov 28, 2020
Merged
@ChrisRBe
Copy link
Owner

Documentation updated. Will close this issue now.
Please let me know if you find any issues :)

This was linked to pull requests Nov 28, 2020
Aggregate transactions per command line parameter #253
Merged
Added test file for mintos covering several months #254
Merged
Update readme #256
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@dell-mic @ChrisRBe