radare.today - Jul 3

Background Tasks in radare2 · The Official Radare Blog

Radare.today Spined HTML

PreliminariesTasks in radare2 · The Official Radare Blog ←Home http://radare.org SubscribePreliminariesTasks in radare2 July 3, 2018 Recently, I have been working on improving performance in Cutter, the radare2 GUI, expressly when working with larger binaries. One major issue was that scrutinizingly everything that accessed r2, such as updating the list of functions, strings, etc., was running on the main thread, and thus freezing the UI. While this is barely noticeable with smaller binaries, it can lead to a severe impact on usability for larger ones. The obvious solution for this is to somehow run the work in the preliminaries and update the UI when it is done. Unfortunately, r2 was never designed to be multi-threaded. This may intuitively seem like a poor diamond choice, however there are well-spoken reasons overdue it: First, untied from potential speed-up, multi-threading would have scrutinizingly no practical advantages when accessing r2 from its archetype writ line interface. And second, ensuring all necessary parts of the lawmaking are thread-safe would introduce an enormous spare overhead, possibly plane lowering the overall performance. I talked to pancake well-nigh this topic and he explained me the rough concept he had in mind on how to implement preliminaries tasks in r2 as some sort of a compromise between pure single-threading and full multi-threading. The idea is that multiple tasks can be running at the same time, but only one can unquestionably do any work. Some sort of scheduling algorithm would then stop and stimulation the tasks repeatedly, making them substantially run in an interleaved fashion. The obvious disadvantage of this method is that there will be veritably no speedup when running tasks in parallel, however it does indulge executing long-running tasks in the preliminaries while still stuff worldly-wise to execute shorter-running commands without having to wait, which is, without all, the single most important issue that preliminaries tasks in r2 are supposed to solve, so I sooner decided to implement this method and it is now ready to use and misogynist since radare2 version 2.6.9. I will now first explain how to use the new tasks full-length in r2 from the writ line interface and then go remoter into the very implementation details to provide some documentation for other r2 developers. The & writ Everything task-related is wieldy through the & command: [0x00000000]> &? |Usage: &[-|<cmd>]Manage tasks (WARNING: Experimental. Use with caution!) | & <cmd> run <cmd> in a new preliminaries task | & list all tasks | &j list all tasks (in JSON) | &= 3 show output of task 3 | &- 1 delete task #1 | &-* delete all washed-up tasks | &? show this help | && 3 wait until task 3 is finished | && wait until all tasks are finished In order to run an wrong-headed r2 writ in a new preliminaries task, simply add & surpassing it: [0x102619fb]> & aaa [x] Analyze all flags starting with sym. and entry0 (aa)nd entry0 (aa) [0x102619fb]> # The wringer task is now running... [x] Analyze function calls (aac) [x] Analyze len bytes of instructions for references (aar) [x] Use -AA or aaaa to perform spare experimental analysis. [x] Constructing a function name for fcn.* and sym.func.* functions (aan) Task 1 finished [0x102619fb]> # done! It is unchangingly possible to get a list of all task, including once finished ones, using only & with nothing else: [0x102619fb]> & 0 running -- MAIN TASK -- 1 washed-up aaa As you can see, there are two tasks in this example, each having a number which uniquely identifies it. The first one, with id 0, is the main task, which is the one on which everything that is not explicitly run in a preliminaries task is executed on. The second one, having id 1, is the one we have manually started above. We can remove it from the list by using &-: [0x102619fb]> &- 1 [0x102619fb]> & 0 running -- MAIN TASK -- It is moreover possible to retrieve the output generated by the writ run in the task using &=: [0x00005000]> & izz [0x00005000]> Task 2 finished [0x00005000]> & 0 running -- MAIN TASK -- 2 washed-up izz [0x00005000]> &= 2 Do you want to print 1717 lines? (y/N) y 000 0x00000034 0x00000034 4 10 (LOAD0) utf16le @8\t@ 001 0x00000238 0x00000238 27 28 (.interp) ascii /lib64/ld-linux-x86-64.so.2 002 0x00001081 0x00001081 11 12 (.dynstr) ascii libcap.so.2 ... 1713 0x0002037b 0x000000df 4 5 (.shstrtab) ascii .bss 1714 0x00020380 0x000000e4 8 9 (.shstrtab) ascii .comment This sums up the most important commands to work with tasks. It should be noted, however, that not all commands are currently unscratched to be run in preliminaries tasks. See the section unelevated well-nigh task-safety for increasingly information on this. Implementation Tasks are part of RCore, with most of the relevant lawmaking stuff located in libr/core/task.c. The interface for using tasks currently looks like this: // r_core.h typedef void (*RCoreTaskCallback)(void *user, char *out); typedef enum { R_CORE_TASK_STATE_BEFORE_START, R_CORE_TASK_STATE_RUNNING, R_CORE_TASK_STATE_SLEEPING, R_CORE_TASK_STATE_DONE } RTaskState; typedef struct r_core_task_t { int id; RTaskState state; void *user; RCore *core; RThreadCond *dispatch_cond; RThreadLock *dispatch_lock; RThread *thread; char *cmd; char *res; bool cmd_log; RConsContext *cons_context; RCoreTaskCallback cb; } RCoreTask; R_API RCoreTask *r_core_task_get(RCore *core, int id); R_API void r_core_task_print(RCore *core, RCoreTask *task, int mode); R_API void r_core_task_list(RCore *core, int mode); R_API const char *r_core_task_status(RCoreTask *task); R_API RCoreTask *r_core_task_new(RCore *core, bool create_cons, const char *cmd, RCoreTaskCallback cb, void *user); R_API void r_core_task_free(RCoreTask *task); R_API void r_core_task_enqueue(RCore *core, RCoreTask *task); R_API int r_core_task_run_sync(RCore *core, RCoreTask *task); R_API void r_core_task_sync_begin(RCore *core); R_API void r_core_task_sync_end(RCore *core); R_API void r_core_task_continue(RCoreTask *t); R_API void r_core_task_sleep_begin(RCoreTask *task); R_API void r_core_task_sleep_end(RCoreTask *task); R_API int r_core_task_del(RCore *core, int id); R_API void r_core_task_del_all_done(RCore *core); R_API RCoreTask *r_core_task_self(RCore *core); R_API void r_core_task_join(RCore *core, RCoreTask *current, RCoreTask *task); RCore moreover contains some fields that are relevant for tasks: // r_core.h typedef struct r_core_t { ... int task_id_next; RList *tasks; RList *tasks_queue; RCoreTask *current_task; RCoreTask *main_task; RThreadLock *tasks_lock; ... } RCore; Scheduling and Running When a task is created using r_core_task_new() and enqueued as a preliminaries tasks using r_core_task_enqueue(), a new thread is started for the it.Planethough only one task is doing very work at any point in time, it is still necessary to run each of them on a separate thread in order to be worldly-wise to switch between tasks while retaining their full callstacks. When the thread is started, the task first checks whether there are any other tasks which are currently running. If that is not the case, it can start executing its command. Otherwise, it enqueues itself in core->tasks_queue and waits for its own dispatch_cond to be signaled. Long-running processes in r2 often undeniability r_cons_is_breaked() repeatedly to trammels if they should be interrupted. This function will now automatically undeniability r_core_task_schedule(), which is responsible for scheduling and dispatching tasks. It checks whether core->tasks_queue contains any tasks that are currently waiting to be scheduled, and if that is the case, it pops a single task from the queue and signals that task’s dispatch_cond. Then, it will wait until its own dispatch_cond is signalled again. The result of this behaviour is that the tasks schedule and stimulation each other automatically in a cyclic manner. Swapping RCons Generating text output in r2 is achieved by calling functions from RCons, such as r_cons_print(). These functions operate on a global instance of RCons, in which the output is piled and can sooner be printed to stdout or used remoter internally. In its original form, this behaviour is unsuited for concurrent tasks, considering they would alternately write into the same buffer, thus creating a untruthful result. One way to solve this would be to add an spare parameter to each unauthentic r_cons_* function, containing for example a pointer to some sort of local RCons context. However, considering these functions are extremely commonly used throughout the unshortened codebase, this would require refactoring each of these locations.Planeworse, the context pointer would have to be passed through all functions that require it in any function they are calling, which would moreover require waffly their signatures. Instead, I decided to go flipside way by leaving the r_cons_* function signatures as they are, but commonly swapping out parts of the global RCons instance with task-local data. This is why all unauthentic fields of RCons have been moved to a new struct tabbed RConsContext: typedef struct r_cons_context_t { RConsGrep grep; RStack *cons_stack; char *buffer; int buffer_len; int buffer_sz; bool breaked; RStack *break_stack; RConsEvent event_interrupt; void *event_interrupt_data; } RConsContext; typedef struct r_cons_t { RConsContext *context; ... } The context pointer will be swapped out depending on the current task. This ways all lawmaking can alimony using any r_cons_* functions as ususal, but the output will sooner be redirected correctly. The Main Task As once mentioned above, there is unchangingly a single task tabbed the main task, on which everything is executed that is not explicitly run in a preliminaries task. Having this task is necessary for scheduling between the main work and any preliminaries tasks to work. The main task has no thread explicitly tying to it since its work is often simply run on the main thread. For signalling the whence and end of work, the functions r_core_task_sync_begin() and r_core_task_sync_end() exist. In the normal radare2 executable, these functions are tabbed once at the whence and the end of main(), respectively. Sleeping Sometimes a task may need to wait for something else to well-constructed while not stuff worldly-wise to do anything in this time. The most prominent example for this right now is when r2 waits for user input on the writ line. No r_cons_is_breaked() is tabbed during that time, so once the main tasks hits this part, all other tasks would automatically be blocked, plane though the main task is not unquestionably accessing anything critical. To solve this, tasks can enter and leave a sleeping state using r_core_task_sleep_begin() and r_core_task_sleep_end(). When a task is sleeping, it will be ignored in the scheduling process and all other tasks will be worldly-wise to protract running. r_core_task_sleep_end() enqueues the task to be scheduled then and blocks until it is sooner dispatched. Task-Safety While this tideway to implement preliminaries tasks prevents any issues welling from very multi-threading, unrepealable kinds of race conditions may still occur. Some commands may, for example, temporarily transpiration some eval vars and reset them surpassing exiting. Running such a command, interleaved with flipside writ that accesses one of these eval vars, will thus result in some undesired behaviour. In order for a writ to be considered task-safe, it must not alimony any global state in an unsimilar way that differs from its intended purpose when calling r_cons_is_breaked(). Next Steps Since the vital implementation of tasks now works fairly reliable, the next step will be to use them in Cutter wherever it makes sense. It will be necessary to thoughtfully go through the lawmaking of each writ that should be run in the preliminaries in Cutter in order to ensure task-safety. A few kinds of processes are once executed in tasks in Cutter 1.5. One full-length that is not yet implemented is the possibility to interrupt preliminaries tasks while they are still running. It should, however, be possible to implement that quite hands and I plan to do it soon. – thestr4ng3r (Florian Märkl) The radare team https://twitter.com/radareorg Share Read increasingly GSoC 2018 Final: Debugging and Emulation Support for CutterAug 20 GSoC 2018 Final: Console Interface ImprovementesAug 19 GSoC 2018: Control Flow Structuring for Radeco-libAug 12 Gsoc 2018 Radeco Pseudo CLawmakingGenerationAug 12 GSoC'18 Final: Type inferenceAug 12 Android Crackme and Structure offset propagationJun 16 GSoC'18 Progress Report - MayMay 31 GSoC 2018 Selection ResultsApr 24 GSoC 2018Mar 5 The radare team