Any questions?

• Question about Task 4 on MP5 led to a review of how to read an evaluation rule, and what the rule in question is actually saying, and what the Task 4 is asking for.
  Consider an evaluation rule like:

  eqn₁
  eqn₂
  ...
  eqn_n
  ------------------------------------
  eqn_consequent
  

  You read such a rule by saying: "how can I justify the consequent equation on the bottom? I justify it by first finding some way to justify the antecedent equations on the top of the line. If I can justify those using the rules, then I can justify the consequence here."
  Task 4 in MP5 is asking you to take the current rule we have for determining the meaning of difference expressions in the EXPLICIT-REFS language (see the evaluation rules given in lecture 6), and replacing it with a similar rule for evaluating difference expressions.
  So one strategy for solving this problem is to compare the original rule with the new one, isolate the places where it is different, and then figure out what the significance of those differences is.
  Once you understand the significance of the differences, and if you understand the meaning of our store passing specification, then you should be able to construct a simple program that illustrates the difference between the EXPLICIT-REFS language and the REFS-EXPLETIVE language.
• A question about EoPL 3.23 led to a discussion of

  let f = proc (x) (x x)
  in (f f)
  

  Felix in mid-discussion revised this to look like

  let f = let y = -(5,2)
          in proc(x) (x x)
  in (f f)
  

  We then did a step by step review of how the interpreter actually evaluates such code in the PROC language.

The b_add transformation

(This is an interactive exercise exploring details of the EXPLICIT-REFS language and the implications of reasoning about programs that modify the Store.)

You can find Ben's transformation here: b-add-transform.

The IMPLICIT-REFS language

Our goal in exploring the IMPLICIT-REFS language is to understand how a language designer might choose to package common patterns by tailoring a language around those patterns.

In EXPLICIT-REFS, there is a lot of notational overhead in allocating locations in the store and referring back to them.

For example, a program that accumulates a value in a counter might have code like:

let tally = newref(0)
in let incr = proc(x)
              begin setref(tally, -(deref(tally),-(0,x)));
                    deref(tally)
              end
   in ...

Some programmers prefer a simpler way to allocate and refer to mutable storage.

To achieve this, we are going to make a new, different extension of the LETREC language. We will break away from our former pattern of making the Denoted Values equivalent to the Expressed Values.

In our new language, IMPLICIT-REFS, every variable will denote a reference to a location in the store.

Such references will not be first-class values; we will not, for example, be able to make a procedure that directly produces a reference as its result. We will also not be able to store references directly into the Store itself; we will only be able to store values such as integers, booleans, and procedures.

Question: what was our representation for the domain of references in the EXPLICIT-REFS interpreter? (No one knew, so we left it aside.)

How about deref? Well, now every variable is associated with a reference, and the only way to get a reference is via a variable, so we can tie location dereferencing to variable lookup. That is, now evaluation of a variable expression will result in both a lookup in the environment ρ for the reference associated with the variable and a lookup in the store σ for the value associated with the location; a "two-level" lookup for the value associated with the variable.

How about setref? We cannot just do away with this the same way we did newref and deref; mutating the store is a fundamental operation that we want to keep (otherwise what's the point of adding a store).

Okay, lets see about implementing this language.

Step-by-Step Illustration of Implementing a Language Extension

1. The first thing we do is take our base language, LETREC and make a copy of the directory holding its interpreter. (Remember, we are not extending the EXPLICIT-REFS language; we are instead making a different extension of the LETREC language that also adds a Store to it.)
2. The way Felix programs is to write his tests first.
   • What should this program result in:

     let x = 3 in begin set x = 4; x end
     

     (That's pretty easy: 4)

   • What should this program result in:

     let x = 3
     in let f = proc (y) x
        in begin set x = 4; (f 5) end
     

     (A little harder. Some people said 3, some said 4. Why is it 4? IMPLICIT-REFS is a tricky language!)

   • What should this program result in:

     let x = 3
     in let f = proc (y) begin set y = 4; x end
        in (f x)
     

     (Who knows? Maybe it should be 4. Or maybe it should be 3. It depends on whether the modification to y also changes x, which depends on whether we are using call-by-value or call-by-reference. (See the book.) We will use call-by-value, so this should result in 3.)

3. Now that we have examples, we encode them as tests in the file tests.scm.

     (define test-list
       '(...
         (simple-set-1 "let x = 3 in begin set x = 4; x end" 4)
         (simple-set-2 "let x = 3
                        in let f = proc (y) x
                           in begin set x = 4; (f 5) end"    4)
         (simple-set-3 "let x = 3
                        in let f = proc (y) begin set y = 4; x end
                           in (f x)"                                3)
         ...))
   

4. Then we switch to the top.scm module and hit the "Run" button.
   Here's the result:

   incorrect answers on tests: (simple-set-1 simple-set-2 simple-set-3)
   

   Inspecting the messages further, we find out that there was a parsing error. This is unsurprising; we have not done anything except write tests yet!
5. So now we add the new grammatical production to the lang.scm module.

     (define the-grammar
       '((...
   
         (expression
          ("set" identifier "=" expression)
          set-exp)
   

6. Hit the run button again.

   interp.scm:25:6: cases: missing cases for the following variants: set-exp in: ...
   

   Again, unsurprising; we added new kinds of abstract syntax to the language, so the cases expressions that dispatch on which kind of abstract syntax trees they encounter need to be updated.
7. So lets put the necessary case into place in interp.scm. We do not know how to implement this code, but we can put some code in as a place holder that will immediately error if the code is ever executed.
   The reason to do this is that we want to verify that at least our parser works, without worrying yet about implementing the rest of the code.
   So we add the following to interp.scm

     (define value-of
       (lambda (exp env)
         (cases expression exp
           ...
           (set-exp (id exp)
              (eopl:error 'set-exp "unimplemented"))
   

8. Then we switch to the top.scm module and hit the "Run" button.

   incorrect answers on tests: (simple-set-1 simple-set-2 simple-set-3)
   

   Hmm. What's wrong now?
   Some careful examination of the error message and interactive experimentation indicates that we were too ambitious with our original tests, because we thought we were only adding a set form, but we were assuming that we already had a begin form available in the language. Whoops!
9. So we could revise our tests to not use begin. (This is not an absurd option, though it would mean that we would have to be careful about what the order of evaluation is in at least one of our other constructs, and probably all of them. It is better to have a certain special form that is dedicated to the purpose of indicating that evaluation order is significant.)
   Instead, lets just quickly add begin as a feature as well.
10. So we add the following production to lang.scm

      (define the-grammar
        '((...
          (expression
           ("begin" expression (arbno ";" expression) "end")
           begin-exp)
    

11. Then we switch to the top.scm module and hit the "Run" button.

    interp.scm:25:6: cases: missing cases for the following variants: begin-exp in: ...
    

12. Once again, we added new kinds of abstract syntax to the language, so the cases expressions that dispatch on which kind of abstract syntax trees they encounter need to be updated.
    Update interp.scm with another bit of code that will error if it is ever executed.

      (define value-of
        (lambda (exp env)
          (cases expression exp
            ...
            (begin-exp (exp1 exps)
               (eopl:error 'begin-exp "unimplemented"))
    

13. Then we switch to the top.scm module and hit the "Run" button.

    incorrect answers on tests: (simple-set-1 simple-set-2 simple-set-3)
    

    Further inspection reveals that the problem is:

    actual outcome:  begin-exp: unimplemented
    incorrect
    

    Great! This is an error we anticipated.
14. So we should implement begin? NO!
    We should first write basic tests of begin.
    We add the following to test.scm

      (define test-list
        '(...
          (begin-exp-1  "let x = 3 in begin x end"      3)
          (begin-exp-2  "let x = 3 in let y = 4 in begin x; y end" 4)  
    

    (Testing begin properly is a little more subtle than the above, since any implementation that returns the value of its final expression will be accepted by the above tests, but this is more of a sanity-check.)
15. Hit the "Run" button in top.scm; no surprise, our begin tests fail, because we haven't implemented it yet.
16. Now that we have put some tests in, we can put in an implementation of begin in interp.scm

      (define value-of
        (lambda (exp env)
          (cases expression exp
            ...
            (begin-exp (exp1 exps)
              (letrec ((begin-loop ;; : Exp Listof[Exp] -> ExpVal
                         (lambda (exp exps)
                           (cond
                             ((null? exps)
                              (value-of exp env))
                             (else
                              (begin (value-of exp env)
                                     (begin-loop (car exps)
                                                 (cdr exps))))))))
                (begin-loop exp1 exps)))
    

17. Okay, our tests for begin passed.
18. Now we have to move on to the real task of implementing implicit references.
19. First of all, we started with the LETREC interpreter, which does not have an implementation of store.scm. We can just snarf the one from the EXPLICIT-REFS interpreter, since both languages will use the same basic abstraction to represent a Store.
20. Next, we need to change our environment representation to reflect the fact that DenVal is now Ref(ExpVal) and not ExpVal.
    So we change data-structures.scm as follows:

      (define-datatype environment environment?
        (empty-env)
        (extend-env 
          (bvar symbol?)
          (bval reference?)        ;; <--
          (saved-env environment?))
        (extend-env-recursively
          (id symbol?)
          (bvar symbol?)
          (body expression?)
          (saved-env environment?)))
    

    We discover that reference? is unbound in this module, because we need to add a line to (require "store.scm") at the top of the data-structures.scm module.
21. We hit the "Run" bottom in top.scm. Now, lots of things break:

    incorrect answers on tests: (positive-const negative-const 
    simple-arith-1 nested-arith-left nested-arith-right 
    test-var-1 test-var-2 test-var-3 
    if-true if-false if-eval-test-true if-eval-test-false if-eval-test-true-2 
    if-eval-test-false-2 simple-let-1 eval-let-body eval-let-rhs 
    simple-nested-let check-shadowing-in-body check-shadowing-in-rhs 
    apply-proc-in-rator-pos apply-simple-proc let-to-proc-1 nested-procs 
    nested-procs2 y-combinator-1 simple-letrec-1 simple-letrec-2 
    simple-letrec-3 HO-nested-letrecs begin-exp-1 begin-exp-2 
    simple-set-1 simple-set-2 simple-set-3)
    

    Felix was not expecting this many failures.
22. In Wednesday's lecture, Felix now made guesses as to what was wrong, since he already had pointed out that many language features, such as let, procedure application, and letrec, would need to change, and he went to each one and "fixed" them in turn. But the tests never got better.

    What Felix should have done, and eventually did do, is check what tests were failing, and determine why.

    In this case, if you just look at the first test that is failing, positive-const, that is just the program "11". Why would this start failing?
    We don't know, so we try running it interactively in the Interactions Window (aka Read-Eval-Print-Loop, aka REPL):

    > (run "11")
    environments.scm::530: extend-env: bad value for bval field: #(struct:num-val 10)
    

    Clicking on the link that DrScheme provides (but also just by going to line 530 of the file environments.scm), we see exactly where things are going wrong, in environments.scm

      (define init-env 
        (lambda ()
          (extend-env 
           'i (num-val 1)
           (extend-env
            'v (num-val 5)
            (extend-env
             'x (num-val 10)
             (empty-env))))))
    

    In particular, DrScheme highlights the expression (extend-env 'x (num-val 10) (empty-env)) in red.
    The problem is that we are building up an initial environment that does not obey our new rule that environments are solely mappings from Variables to DenVal = Ref(ExpVal).
    This is easily fixed:

      (define init-env 
        (lambda ()
          (extend-env 
           'i (newref (num-val 1))    ;; <--
           (extend-env
            'v (newref (num-val 5))   ;; <--
            (extend-env
             'x (newref (num-val 10)) ;; <--
             (empty-env))))))
    

    That, and a line to (require "store.scm"), and we are ready to try to run our tests again
23. We hit the "Run" bottom in top.scm. Again, lots of things break, but this is more reasonable, because now it is tests that involve variables that are breaking:

    incorrect answers on tests: (test-var-1 test-var-2 test-var-3 simple-let-1 
    eval-let-body eval-let-rhs simple-nested-let 
    check-shadowing-in-body check-shadowing-in-rhs 
    apply-proc-in-rator-pos apply-simple-proc let-to-proc-1 
    nested-procs nested-procs2 y-combinator-1 
    simple-letrec-1 simple-letrec-2 simple-letrec-3 
    HO-nested-letrecs begin-exp-1 begin-exp-2 
    simple-set-1 simple-set-2 simple-set-3)
    

24. So now lets fix some of the stuff we know is going to be wrong: we have to update how we handle let expressions, since now we must allocate a location every time we bind a variable.
    So we change interp.scm:

      (define value-of
        (lambda (exp env)
          (cases expression exp
            ...
            (let-exp (id rhs body)       
              (let ((val (value-of rhs env)))
                (let ((loc (newref val)))
                  (value-of body
                            (extend-env id loc env)))))
    

25. We again discover we have unbound variable newref because we are missing a line to (require "store.scm") in the interp module.
26. We also will need to change our procedure invocation works, because that is a place where a variable binding is added to the environment, and so it will also allocation a location on the store to associate with that variable.
    So change interp.scm as follows: apply-procedure's contract needs to be corrected, and value-of's code needs to be changed:

      ;; apply-procedure : procedure * denval -> expval
    
      (define value-of
        (lambda (exp env)
          (cases expression exp
            ...
            (app-exp (rator rand)          
              (let ((proc (expval->proc (value-of rator env)))
                    (arg  (value-of rand env)))
    	    (let ((loc (newref arg)))       ;; <--
                  (apply-procedure proc loc)))) ;; <--
    

27. How about letrec?
    • We had a detour here on Wednesday to review the trick EOPL is using to implement letrec by creating the closures on demand, during variable lookup.
    • After a thorough explanation of the trick, proceeded to say, "Okay, so since we're already creating the closures on demand every time the recursive function is looked up, lets do the same thing and just allocate the location on demand every time the recursive function is looked up."
    • I must stress that I am just following the book on this point; this strategy may or may not be the right thing to do, and the students should explore the behavior of this version, and try to understand the design space for what different behavior might be desirable.
    So, after that detour, here's the change to environments.scm:

      (define apply-env
        (lambda (env search-sym)
          (cases environment env
             ...
             (extend-env-recursively (id bvar body saved-env)
               (if (eqv? search-sym id)
                 (newref (proc-val (procedure bvar body env)))  ;; <-- 
                 (apply-env saved-env search-sym))))))
    

28. Hit the "Run" button in top.scm... but again everything fails. Why?
29. The answer is because we forgot to handle the new two-level lookup for variable references.
    If we had been smart, we would have more closely inspected the tests that were failing, because some of them test this directly and have nothing to do with introducing new bindings via let, proc, or letrec.
    Anyway, here is the implementation of the two-level lookup in interp.scm

      (define value-of
        (lambda (exp env)
          (cases expression exp
            ...
            (var-exp (id) (deref (apply-env env id)))
    

30. Hit the "Run" button in top.scm...

    incorrect answers on tests: (simple-set-1 simple-set-2 simple-set-3)
    

    At this stage, the only tests that fail are the ones for the feature we just added; no regressions! (Ship it to the client!)
31. Okay, so was all this work for naught?
    Look at the tests that are failing: they all say:

    actual outcome:  set-exp: unimplemented
    

    We laid the ground work we needed to actually implement set-exp expressions, but forgot to actually put the code in for them!
    Here's the change to interp.scm

      (define value-of
        (lambda (exp env)
          (cases expression exp
            ...
            (set-exp (id exp)
               (let ((val (value-of exp env))
                     (loc (apply-env env id)))
                 (begin (setref! loc val)
                        (num-val 23))))
    

32. With that final change, hit the "Run" button in the top.scm module.

    no bugs found
    

    set expressions "work"!

Or do they...?

Any last questions?

• Is there a midterm next week?
  Yes! The monday section must attend the monday midterm, and the wednesday section must attend the wednesday midterm.

Last updated 22 February 2008.