Effective Python: Projects

programming
Python
effective python
Author

Stephen J. Mildenhall

Published

2022-04-05

The best way to learn Python is to use it to solve a real-life problem you face. I’m often asked to suggest example projects that are good for beginners. That’s a tricky question because we all have different issues and interests. Rather than recommend contrived problems, here is a partial list of how I’ve used Python since I started programming in 2016.

Going out on a limb, if I had to pick one for an actuary, it would be number 19. If you ever have to pick up and use a new database, a generic explorer tool is very helpful.

Please write me at and let me know how you’ve used Python in your work or leisure.

Python Projects since 2016

  1. Aggregate: tools for creating aggregate loss distributions. Aggregate was used to generate all the examples and figures in my forthcoming book, written with John Major. There is even (Python generated) documentation.
  2. Document manipulation tools to build presentations and websites from Markdown called GREAT Generalizable, Reusable, Extensible Actuarial Toolkit (formally a set of VBA macros). GREAT functions were used to create all PDF the files in PIRC. There is partial documentation, again created automatically from the Python source code. (Yes, you can do that in R too.)
  3. Websites: my websites are all written in Python, using Flask and jinja. For example, my blog is run from a Pandas dataframe using GREAT to create the html from Markdown.
  4. Regular expressions for text analysis. E.g., solutions to the Wordle word puzzle on my website.
  5. Webscraping: Collect information from websites and aggregate into a useful database. We will partially cover this in sessions 4, or 5. Examples include an early COVID tracker, a database of start-ups, and lots of BEA, FRED related analysis.
  6. Implementing the Iman-Conover method for correlation using numpy, session 5.
  7. Simulating random random correlation matrices using vines, session 5.
  8. Implementing the Rearrangement Algorithm for worst-case VaR performance, session 5.
  9. Building Dashboards, for example, an automated system to analyze subreddits. We’ll cover graphics later too.
  10. Making word clouds, including extracting stop words and stemming. See the left-hand side of the Actuary reddit dashboard
  11. Exploring the Bitcoin blockchain.
  12. Converting between graphics file types and manipulating PDFs.
  13. Making cartograms (my current project), such as this analysis of the 2020 Presidential Election. This project leverages geopandas, which provides a full mapping and visualization querying language and toolkit.
  14. Playing with open APIs such as flickr, news, reddit, twitter, Evernote, fitbit, Google Scholar, Wikipedia. E.g., the background image quit in my LinkedIn profile was assembled from images scraped from Google books.
  15. Organizing files, e.g., finding duplicate files, and creating links to presentations by client and by file type.
  16. Data analysis based on SNL/S&P Market Intelligence downloads, all done in Pandas.
  17. General text re-arrangement, extracting information from free-form text. E.g., accurately counting the number of words in my book, which turned out to be surprisingly difficult.
  18. Miscellaneous nerdy math stuff, such as testing for primes, computing the CAS Nonce using parallel processing, and evaluating \(2^{402653211}\) exactly (yes, that number comes up).
  19. Database manipulation and exploration: building a generic wrapper around a database to facilitate metadata querying: show me all the tables, their constructor, the queries, indexes, field, return the top 10 rows in each, etc. I used it to explore and reverse engineer my Evernote SQLite database.
  20. Tracking my investments, building near real-time price downloads and analyzing historical price, risk and performance.
  21. Extensions to SublimeText, which is the editor I use. ST is written in Python and you can write Python extensions. Mine are workflow and formatting related.
  22. Computing 1000! exactly. Yes, it can do that. The answer equals 402387260077093773543702433923003985719374864210714632543799910429938512398629020592044208486969404800479988610197196058631666872994808558901323829669944590997424504087073759918823627727188732519779505950995276120874975462497043601418278094646496291056393887437886487337119181045825783647849977012476632889835955735432513185323958463075557409114262417474349347553428646576611667797396668820291207379143853719588249808126867838374559731746136085379534524221586593201928090878297308431392844403281231558611036976801357304216168747609675871348312025478589320767169132448426236131412508780208000261683151027341827977704784635868170164365024153691398281264810213092761244896359928705114964975419909342221566832572080821333186116811553615836546984046708975602900950537616475847728421889679646244945160765353408198901385442487984959953319101723355556602139450399736280750137837615307127761926849034352625200015888535147331611702103968175921510907788019393178114194545257223865541461062892187960223838971476088506276862967146674697562911234082439208160153780889893964518263243671616762179168909779911903754031274622289988005195444414282012187361745992642956581746628302955570299024324153181617210465832036786906117260158783520751516284225540265170483304226143974286933061690897968482590125458327168226458066526769958652682272807075781391858178889652208164348344825993266043367660176999612831860788386150279465955131156552036093988180612138558600301435694527224206344631797460594682573103790084024432438465657245014402821885252470935190620929023136493273497565513958720559654228749774011413346962715422845862377387538230483865688976461927383814900140767310446640259899490222221765904339901886018566526485061799702356193897017860040811889729918311021171229845901641921068884387121855646124960798722908519296819372388642614839657382291123125024186649353143970137428531926649875337218940694281434118520158014123344828015051399694290153483077644569099073152433278288269864602789864321139083506217095002597389863554277196742822248757586765752344220207573630569498825087968928162753848863396909959826280956121450994871701244516461260379029309120889086942028510640182154399457156805941872748998094254742173582401063677404595741785160829230135358081840096996372524230560855903700624271243416909004153690105933983835777939410970027753472000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.