Homework 3 Due: Wednesday, Oct. 28, 2015 (two weeks and a day) [ NOTE: I have added: /course/cs5600f15/homework/hw3-comments and: /course/cs5600f15/homework/hw3-relinking-libckpt I am collecting information from the students about this homework, and I am offering it here, to share information. ] [ NOTE: I have decided that it is enough to do any one of the three parts below, _and_ also to write a document discussing what are the issues to overcome and how you would do so, in attempting the other two parts. To summarize, it suffices to submit code for one of the three parts, but you must still discuss the other two parts in a document. (And of course, your document should also discuss the part for which you _did_ produce code. ] Copyright (c) Gene Cooperman, 2015. Full rights to copy, modify, and re-distribute this text is given, provided that this copyright notice remains. No warranty is provided. This homework builds on hw2 (mini-DMTCP). In this assignment, you will explore several other operating system concepts: * Tracing a process (ptrace) - The debugger GDB is built on top of 'ptrace'. * Library injection * Dynamic shared libraries * Further loader features: LD_PRELOAD, LD_DEBUG (man ld.so) * Trampolines * dlopen, dlsym: direct calls to * Wrappers around library functions (dlopen, dlsym) - Wrapper functions in the area of "systems" are the equivalent of the Adapter pattern in software engineering - Wrappers are also a key to operating system abstraction layers (OSAL). An OSAL is often used in embedded or real-time systems, but can also be used elsewhere (for example, the DMTCP plugins): https://en.wikipedia.org/wiki/Operating_system_abstraction_layer This homework is in three parts: A. Attaching libckpt.a to a statically linked running process B. Adding dynamic libraries to a statically linked process C. Building a plugin (a wrapper function) ======================================================================= A. Attaching libckpt.a to a statically linked running process [ Please be sure to read: /course/cs5600f15/homework/hw3-relinking-libckpt as important background for doing this part. ] Using ptrace (man ptrace), we will inject the libckpt.a library into a running statically linked process. You will then detach from the running process. You now have the equivalent of your implementation of mini-DMTCP. After this, checkpoint and restart should work, exactly as it did in hw2 (in mini-DMTCP). To do this you will need to look up the following options in 'man ptrace': PTRACE_ATTACH, PTRACE_POKETEXT, PTRACE_ATTACH Note that there is a tutorial on ptrace here: http://www.linuxjournal.com/article/6100 There are still three other issue that you will face. 1. Your code, libckpt.a, may be position-indpendent, or it may not. If it is not position-independent, it expects that the loader will patch the code in RAM, so that any absolute addresses are modified according to where it was actually loaded. The flags "-Wl,-Ttext=" and "-Wl,-Tdata=" for the loader in hw2 are too late to instruct the compiler to issue position-independent code. To instruct the compiler to issue position-independent code, try: gcc -o libckpt.a -fPIC -DPIC ... 2. Because we didn't use the traditional loader to inject libckpt.a, our constructor function is not going to be executed. So, we will not declare our SIGUSR2 handler to the kernel. To fix this, we must call the constructor manually from our attaching process using ptrace. So, we want to force the target process to call this function. The easiest way to do this may be to temporarily save all registers, and then reset the program counter register. One way to accomplish this is to write a function libckpt_init() inside libckpt.a, and to ensure that it takes no arguments. (Otherwise, we would have to pass its arguments in other registers.) The function libckpt_init() should call your constructor function, and when it returns, it should enter an infinite loop. Upon return, you can then restore the original registers. For this part, you will want to use: PTRACE_GETREGS and PTRACE_SETREGS 3. We can only use PTRACE_POKETEXT on memory that has already been mapped in. There are two alternatives. The simple alternative (recommended for this exercise) is to modify the target application to allocate a very large memory region, and to change the protections to 'rwx'. Then print the address of the beginning of that region to the screen. Finally, your attaching process can then "poke" libckpt.a to the newly available RAM. 4. You may or may not have permission to modify the target application. If you don't have permission, give yourself permission by modifying the target application itself. See "REMARK 2" below. REMARK 1: There are several alternatives to having the target application. One is to use PTRACE_PEEKTEST copy out some original memory in the text segment, and then copy in some small code that will call "mmap()" to allocate the new storage. Using /proc/*/maps, you can find the original text segment where you will replace memory. Then read the old program counter register, and set the program counter register to point to the new code. Finally, after executing, copy the original code back in, and reset the original text segment. Another is a variation on this theme, is found inside: http://stackoverflow.com/questions/24355344/inject-shared-library-into-a-process Go to the section "Consider this trivial program, compiled as say target:". REMARK 2: On most Linux distros, they will allow you to use ptrace only on a child process. This is often a problem for 'gdb attach' and for our 'attach' exercies. There are several ways to get around this. If this affects you, a simple workaround is to add: prctl(PR_SET_PTRACER, PR_SET_PTRACER_ANY); to the target application that will be attached to. Another workaround is to do the attach as root. A third workaround is to permanently change the default by running as root the commnet: echo 0 > /proc/sys/kernel/yama/ptrace_scope ======================================================================= B. Adding dynamic libraries to a statically linked process BACKGROUND: The way that dynamically linked libraries are implemented in GNU are through ld-linux.so. (To see the actual name of this library on your system, do: "ldd /bin/ls".) When gcc compiles a dynamically linked executable, it uses a _start function that calls _dl_start. To verify this, do: gdb /bin/ls (gdb) break _dl_start (gdb) run (gdb) where The library libdl.so defines dlopen and dlsym. To verify this, either do: 'man dlopen' and note that the man page says that it requires '-ldl', or else directly do: nm -D libdl.so [ You'll have to replace 'libdl.so' by the 'libdl.so' on your sysgtem. As before, you can find it through "ldd /bin/ls". ] However, note that the loader, ld.so (found locally on our system as /lib64/ld-linux-x86-64.so.2 ) is the function that defineds _dl_start. So, the first step will be to inject /lib64/ld-linux-x86-64.so.2 . Note also that we found /lib64/ld-linux-x86-64.so.2 in either of two ways: 1. by doing: ldd /bin/ls 2. or by doing: vi /bin/ls and looking near the header of the file for the ELF interpreter. Note also that "ldd /lib64/ld-linux-x86-64.so.2" tells us that this library does not depend on any other libraries. This is good for what we are attempting. Next, instead of injecting libckpt.a, you will inject libdl.so, following the same recipe as before. Note that dynamically linked libraries are always compiled as position-independent code. So, that's one less thing to worry about. In the case of libckpt.a, we called it by calling our constructor function, which then called 'signal()' to declare the SIGUSR2 signal handler. In the case of 'libdl.so', we want to call _dl_start(). Use the same techniques to call _dl_start(). This will require some detective work, similar to hw1. As before, we will use "gdb". Our goal is to find the memory address of '_dl_start()' within the text segment, and we will then apply that same offset to wherever the text segment is loaded inside your target process to be attached. Also, not that if you inspect the _start call frame within GDB using the assembly directives (see hw2), you will then find that there is a "part 2" to '_dl_start()', called '_dl_start_user()'. You may alternatively need to call '_dl_start_user()'. Luckily, the same approach as above will find the address of '_dl_start_user()' for you. Finally, in order to test your program, try: LD_PRELOAD=libckpt.so attacher ./hello libdl.so should look at LD_PRELOAD and do the right thing. See 'man ld.so'. ======================================================================= C. Building a plugin (a wrapper function) We wish to create a wrapper function for a function in libc.a. You must demonstrate by placing a wrapper function around sleep(), similarly to the sleep wrapper function in DMTCP: test/plugin/sleep1 We've seen how we can inject a checkpoint library into a running library. We've also seen how to inject the ability to do dynamic linking into a running process. There are two ways that you might do this (or a third way, if you find one). One way is to patch the actual libc.a. If you find the entry point of the function, sleep(), in libc.a, you can modify the code at assembly level to inject assembly code that jumps to a new location, executes the code before the wrapper, returns to the sleep() function for full execution, and then returns to a post-sleep function to finish the wrapper function. Another method is to patch vdso or vsyscall itself. As an alternative, if you can inject libdl.so, then you can simply patch libc.a to call a wrapper function, that wraps around the sleep() function in libc.so. Finally, if you can inject libdl.so, you can use LD_PRELOAD to preload a wrapper function, libplugin-sleep.so, followed by libc.so: LD_PRELOAD=libplugin-sleep.so:libc.so You are welcome to explore other alternatives. Putting it all together (extra credit): DMTCP attach: ptrace: inject libdl.so, init. [ PRELOAD gets libckpt_init:libpulign-sleep.so:libc.so libckpt_init.so can patch libc.a of user ] Set env. var. LD_PRELOAD=libckpt_init.so in advance (or add it with ptrace) Make this work for a dynamically linked program instead of a statically linked program. This requires you to place libdkckpt.so in fromt of all other processes (just as we did LD_PRELOAD=libplugin-sleep.so:libc.so in step C.) To make this work, you will need to remove the previous libc.so of a dynamically linked program. Luckily, dlclose() allows you to do this, if you can find the handle of the current libc.so, which now exists in memory of the running process.)