#180 Transactional file IO with Python and safer

Python Bytes - A podcast by Michael Kennedy and Brian Okken - Mondays

Podcast artwork

Categories:

Technology

Sponsored by DigitalOcean: pythonbytes.fm/digitalocean - $100 credit for new users to build something awesome.

Michael #1: Ubuntu 20.04 is out!

  • Next LTS support version since 26th April 2018 (18.04).
  • Comes with Python 3.8 included!
  • Already upgraded all our servers, super smooth.
  • Kernel has been updated to the 5.4 based Linux kernel, with additional support for Wireguard VPN, AUFS5, and improved support for IBM, Intel, Raspberry Pi and AMD hardware.
  • Features the latest version of the GNOME desktop environment.
  • Brings support for installing an Ubuntu desktop system on top of ZFS.
  • 20.04 already an option on DigitalOcean ;)

Brian #2: Working with warnings in Python

  • (Or: When is an exception not an exception?)
  • Reuven Lerner
  • Exceptions, the class hierarchy of exceptions, and warnings.
  • “… most of the time, warnings are aimed at developers rather than users. Warnings in Python are sort of like the “service needed” light on a car; the user might know that something is wrong, but only a qualified repairperson will know what to do. Developers should avoid showing warnings to end users.”
  • Python’s warning system …:
    • It treats the warnings as a separate type of output, so that we cannot confuse it with either exceptions or the program’s printed text,
    • It lets us indicate what kind of warning we’re sending the user,
    • It lets the user indicate what should happen with different types of warnings, with some causing fatal errors, others displaying their messages on the screen, and still others being ignored,
    • It lets programmers develop their own, new kinds of warnings.
  • Reuven goes on to show how to use warnings in your code.
    • using them
    • creating custom warnings
    • filtering

Michael #3: Safer file writer

    with open(filename, 'w') as fp:
        json.dump(data, fp)
  • It’s using with, so it’s good right?
  • Well the file itself may be overwritten and maybe corrupted
  • With safer, you write almost identical code:
with safer.open(filename, 'w') as fp:
    json.dump(data, fp)

Brian #4: codespell

  • codespell : Fix common misspellings in text files. It's designed primarily for checking misspelled words in source code, but it can be used with other files as well.
  • I got a cool pull request against the cards project to add a pre-commit hook to run codespell. (Thanks Christian Clauss)
  • codespell caught a documentation spelling error in cards, where I had spelled “arguments” as “arguements”. Oops.
  • Spelling errors are annoying and embarrassing in code and comments, and distracting. Also hard to deal with using traditional spell checkers. So super glad this is a thing.

Michael #5: Austin profiler

  • via Anthony Shaw
  • Python frame stack sampler for CPython
  • Profiles CPU and Memory!
  • Why Austin?
    • Written in pure C Austin is written in pure C code. There are no dependencies on third-party libraries.
    • Just a sampler - fast: Austin is just a frame stack sampler. It looks into a running Python application at regular intervals of time and dumps whatever frame stack it finds.
    • Simple output, powerful tools Austin uses the collapsed stack format of FlameGraph that is easy to parse. You can then go and build your own tool to analyse Austin's output.
    • You could even make a player that replays the application execution in slow motion, so that you can see what has happened in temporal order.
    • Small size Austin compiles to a single binary executable of just a bunch of KB.
    • Easy to maintain Occasionally, the Python C API changes and Austin will need to be adjusted to new releases. However, given that Austin, like CPython, is written in C, implementing the new changes is rather straight-forward.
  • Creates nice flame graphs

  • The Austin TUI is nice! Austin TUI

  • Web Austin is yet another example of how to use Austin to make a profiling tool. It makes use of d3-flame-graph to display a live flame graph in the web browser that refreshes every 3 seconds with newly collected samples.

  • Austin output format can be converted easily into the Speedscope JSON format. You can find a sample utility along with the TUI and Austin Web.

Brian #6: Numbers in Python

  • Moshe Zadka
  • A great article on integers, floats, fractions, & decimals
  • Integers
    • They turn into floats very easily, (4/3)*3 → 4.0, int → float
  • Floats
    • don’t behave like the floating point numbers in theory
    • don’t obey mathematical properties
      • subtraction and addition are not inverses
        • 0.1 + 0.2 - 0.2 - 0.1 != 0.0
      • addition is not associative
    • My added comment: Don’t compare floats with ==, use pytest.approx or other approximation techniques.
  • Fractions
    • Kinda cool that they are there but be very careful about your input
    • Algorithms on fractions can explode in time and to some extent memory.
    • Generally better to use floats
  • Decimals
    • Good for financial transactions.
    • Weird dependence on a global state variable, the context precision.
    • Safer to use a local context to set the precision locally
    >>> with localcontext() as ctx:
    ...     ctx.prec = 10
    ...     Decimal(1) / Decimal(7)
    ...
    Decimal('0.1428571429')

Extras:

Brian:

Michael:

Joke:

Unix is user friendly. It's just very particular about who its friends are. (via PyJoke)

If you put 1000 monkeys at 1000 computers eventually one will write a Python program. The rest will write PERL. (via @JamesAbel)